Sep 16 12:01:50.232834 ip-10-0-136-170 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:01:50.673187 ip-10-0-136-170 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:01:50.673187 ip-10-0-136-170 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:01:50.673187 ip-10-0-136-170 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:01:50.673187 ip-10-0-136-170 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:01:50.673187 ip-10-0-136-170 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:01:50.675023 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.674931    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:01:50.678101 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678085    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:50.678101 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678101    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678105    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678108    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678112    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678114    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678117    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678121    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678126    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678129    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678132    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678135    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678139    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678142    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678144    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678147    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678149    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678151    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678154    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678156    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:50.678168 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678159    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678162    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678165    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678167    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678170    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678173    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678176    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678179    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678185    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678188    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678190    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678192    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678195    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678197    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678200    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678202    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678204    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678207    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678209    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678212    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:50.678665 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678214    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678217    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678219    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678221    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678224    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678228    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678233    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678236    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678239    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678242    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678244    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678247    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678250    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678253    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678256    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678258    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678261    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678264    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678267    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678270    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:50.679146 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678272    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678275    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678278    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678280    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678283    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678286    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678289    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678291    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678294    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678296    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678298    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678302    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678306    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678308    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678311    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678313    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678316    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678318    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678322    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678324    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:50.679645 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678327    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678330    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678333    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678335    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678337    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678340    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678343    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678346    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678348    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678350    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678353    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678356    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678358    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678361    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678363    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678366    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678368    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678371    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.678373    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:50.680116 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679085    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679092    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679095    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679098    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679101    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679104    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679106    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679109    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679112    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679115    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679118    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679121    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679123    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679126    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679129    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679132    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679135    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679137    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679140    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679144    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:50.680595 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679146    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679149    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679152    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679154    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679157    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679160    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679162    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679165    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679167    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679170    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679172    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679175    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679177    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679180    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679182    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679184    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679187    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679189    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679192    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679196    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:50.681095 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679200    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679203    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679207    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679212    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679216    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679219    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679221    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679225    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679227    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679230    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679234    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679238    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679241    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679244    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679247    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679250    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679252    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679255    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679257    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:50.681687 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679260    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679262    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679265    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679268    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679270    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679272    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679275    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679277    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679280    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679283    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679285    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679288    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679291    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679293    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679296    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679298    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679301    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679304    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679307    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:50.682159 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679309    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679312    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679316    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679318    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679321    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679323    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679326    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679328    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679331    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679333    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679336    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679338    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679340    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679343    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679345    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679348    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679350    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679353    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679355    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679357    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:50.682666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679360    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679750    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679756    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679759    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679762    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679764    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679767    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679769    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679772    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679775    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679779    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679783    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679785    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679788    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679791    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679794    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679796    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679798    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679848    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679852    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:50.683150 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679856    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679859    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679862    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679867    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679871    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679876    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679880    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679884    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679888    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679891    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679896    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679899    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679902    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679904    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679909    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679911    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679914    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679916    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679918    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679921    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:50.683673 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679924    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679927    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679929    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679932    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679935    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679937    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679940    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679944    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679947    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679950    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679952    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679954    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679957    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679960    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679962    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679965    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679967    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679969    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679972    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679974    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:50.684184 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679978    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679980    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679983    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679986    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679988    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679990    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679994    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679997    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.679999    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680002    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680004    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680007    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680009    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680012    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680015    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680017    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680020    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680022    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680025    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680029    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:50.684705 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680033    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680036    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680039    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680041    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680044    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680047    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680049    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680052    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680054    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680056    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680059    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680062    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680065    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680068    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680071    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680074    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680077    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680079    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680083    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680085    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:50.685220 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680168    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680212    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680223    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680228    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680233    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680236    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680241    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680245    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680248    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680251    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680254    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680257    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680260    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680264    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680267    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680271    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680273    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680276    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680280    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680283    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680286    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680289    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680293    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680296    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:01:50.685742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680299    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680303    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680306    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680309    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680314    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680316    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680320    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680324    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680328    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680331    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680334    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680337    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680340    2578 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680343    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680348    2578 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680351    2578 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680354    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680357    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680360    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680364    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680367    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680370    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680373    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680376    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680379    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:01:50.686330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680382    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680384    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680387    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680390    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680393    2578 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680397    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680400    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680403    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680406    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680409    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680412    2578 flags.go:64] FLAG: --help="false"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680416    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-170.ec2.internal"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680419    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680422    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680425    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680429    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680432    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680435    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680438    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680441    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680444    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680447    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680450    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680453    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:01:50.686964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680456    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680458    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680461    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680464    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680467    2578 flags.go:64] FLAG: --lock-file=""
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680470    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680473    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680476    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680482    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680485    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680488    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680491    2578 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680494    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680497    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680500    2578 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680502    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680507    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680522    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680527    2578 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680531    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680534    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680537    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680540    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680543    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680546    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:01:50.687610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680549    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680557    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680559    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680562    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680566    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680569    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680573    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680576    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680579    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680582    2578 flags.go:64] FLAG: --port="10250"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680585    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680588    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ffe6f1e392a46f1b"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680591    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680594    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680597    2578 flags.go:64] FLAG: --register-node="true"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680601    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680604    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680607    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680610    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680613    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680617    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680620    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680623    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680626    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680629    2578 flags.go:64] FLAG: --runonce="false"
Sep 16 12:01:50.688240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680631    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680635    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680639    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680642    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680645    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680648    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680651    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680654    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680657    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680660    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680663    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680666    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680669    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680672    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680674    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680680    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680683    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680686    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680689    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680693    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680695    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680698    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680702    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680705    2578 flags.go:64] FLAG: --v="2"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680709    2578 flags.go:64] FLAG: --version="false"
Sep 16 12:01:50.688900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680713    2578 flags.go:64] FLAG: --vmodule=""
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680716    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.680720    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680835    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680839    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680842    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680844    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680847    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680849    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680853    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680857    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680860    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680863    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680865    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680868    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680871    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680873    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680876    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680879    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680882    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:50.689504 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680886    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680889    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680892    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680894    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680897    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680900    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680903    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680905    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680908    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680910    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680913    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680916    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680919    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680921    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680924    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680927    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680929    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680933    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680937    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:50.690014 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680939    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680943    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680947    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680951    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680954    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680956    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680960    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680963    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680965    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680968    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680986    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680990    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680994    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.680997    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681000    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681004    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681007    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681010    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681013    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:50.690476 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681016    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681018    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681021    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681024    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681027    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681030    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681033    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681036    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681039    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681041    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681044    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681046    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681049    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681051    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681054    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681059    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681063    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681066    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681068    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681071    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681073    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:50.691001 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681076    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681078    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681081    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681084    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681086    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681089    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681091    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681094    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681097    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681099    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681102    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681105    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681108    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681110    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681113    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681115    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681118    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681121    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681123    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:50.691535 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681126    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681128    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681131    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.681133    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.681786    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.690413    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.690429    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.691214    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.691236    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691331    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691341    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691346    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691351    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691355    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:50.692008 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691359    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691365    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691370    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691374    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691378    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691382    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691386    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691391    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691395    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691400    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691403    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691406    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691409    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691411    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691414    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691417    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691419    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691422    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691424    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691427    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:50.692392 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691429    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691432    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691435    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691437    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691442    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691445    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691448    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691450    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691453    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691455    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691458    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691460    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691464    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691466    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691469    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691471    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691474    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691476    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691479    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691481    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:50.692969 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691484    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691486    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691489    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691491    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691494    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691497    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691499    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691502    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691505    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691507    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691527    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691532    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691538    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691543    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691546    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691550    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691553    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691556    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691559    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:50.693462 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691562    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691565    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691569    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691573    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691576    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691579    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691582    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691584    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691587    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691589    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691592    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691594    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691597    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691600    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691602    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691604    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691608    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691612    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691614    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:50.693941 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691617    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691619    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691622    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691625    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691627    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691630    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691632    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691636    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691638    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691642    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691644    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691647    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691650    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691653    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691655    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:50.694398 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691658    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.691664    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691784    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691789    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691792    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691795    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691799    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691802    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691805    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691808    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691810    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691814    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691817    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691819    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691822    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:50.694778 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691825    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691827    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691830    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691833    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691835    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691838    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691840    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691842    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691845    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691848    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691850    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691853    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691856    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691859    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691861    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691866    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691870    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691873    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691876    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:50.695144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691879    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691882    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691884    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691887    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691890    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691892    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691895    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691897    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691900    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691902    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691905    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691907    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691910    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691912    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691915    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691918    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691920    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691923    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691925    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691927    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:50.695617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691930    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691932    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691935    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691938    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691940    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691943    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691945    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691949    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691953    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691956    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691958    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691961    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691963    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691966    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691968    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691971    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691973    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691976    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691978    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:50.696088 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691981    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691983    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691985    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691988    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691990    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691992    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691995    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.691999    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692002    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692004    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692007    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692010    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692013    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692016    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692019    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692021    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692024    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692027    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692030    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692032    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:50.696555 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692035    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692037    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692039    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692042    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692045    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692047    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692049    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:50.692052    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.692057    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.692182    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.694271    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.695578    2578 server.go:1019] "Starting client certificate rotation"
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.695673    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:01:50.697045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.695707    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:01:50.698367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.698353    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:01:50.699615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.699597    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:01:50.711863 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.711838    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:01:50.714745 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.714729    2578 log.go:25] "Validated CRI v1 image API"
Sep 16 12:01:50.716355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.716342    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:01:50.718832 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.718806    2578 fs.go:135] Filesystem UUIDs: map[1b27413b-9d98-49d2-b573-836c9569fe84:/dev/nvme0n1p4 4913bf78-35bd-498a-8b42-9c1cf5bfc0ca:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 16 12:01:50.718907 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.718831    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:01:50.724660 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.724532    2578 manager.go:216] Machine: {Timestamp:2026-09-16 12:01:50.72319776 +0000 UTC m=+0.360522590 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102778 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f95e60cd5bd317a72e500efb25e00 SystemUUID:ec2f95e6-0cd5-bd31-7a72-e500efb25e00 BootID:cd3cd56b-880f-4375-a19a-3a2675ffbb17 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:1a:aa:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:1a:aa:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:d6:89:71:ad:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:01:50.724660 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.724650    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:01:50.724843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.724770    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:01:50.727555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.727522    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:01:50.727698 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.727558    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:01:50.728238 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.728229    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:01:50.728269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.728241    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:01:50.728269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.728250    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:01:50.728269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.728266    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:01:50.729252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.729242    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:01:50.730154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.730137    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:01:50.733377 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.733363    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:01:50.733528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.733500    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:01:50.733570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.733548    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:01:50.733570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.733560    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:01:50.733627 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.733577    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:01:50.733627 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.733591    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:01:50.733627 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.733603    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:01:50.734020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.734008    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:01:50.734056 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.734028    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:01:50.736628 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.736601    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:01:50.738662 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.738646    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:01:50.738713 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.738679    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:01:50.740345 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740330    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:01:50.740345 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740348    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740354    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740359    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740367    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740372    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740385    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740390    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740403    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740408    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740416    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:01:50.740451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.740422    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:01:50.741778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.741766    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:01:50.741778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.741779    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:01:50.745816 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.745769    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:01:50.745902 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.745846    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-170.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:01:50.745902 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.745853    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:01:50.746217 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.746206    2578 server.go:1268] "Started kubelet"
Sep 16 12:01:50.746269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.746244    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:01:50.746500 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.746417    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:01:50.746635 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.746621    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:01:50.749803 ip-10-0-136-170 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:01:50.751758 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.751717    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:01:50.753952 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.753933    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:01:50.754492 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.754422    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:01:50.758655 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.757270    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-170.ec2.internal.18d5cb70713bdf46  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-170.ec2.internal,UID:ip-10-0-136-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-170.ec2.internal,},FirstTimestamp:2026-09-16 12:01:50.746165062 +0000 UTC m=+0.383489948,LastTimestamp:2026-09-16 12:01:50.746165062 +0000 UTC m=+0.383489948,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-170.ec2.internal,}"
Sep 16 12:01:50.759354 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.759337    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:01:50.759897 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.759880    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:01:50.759993 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.759970    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:50.760653 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.760634    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:01:50.760653 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.760635    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:01:50.760800 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.760661    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:01:50.760890 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.760836    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:01:50.760890 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.760847    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:01:50.761487 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.761325    2578 factory.go:55] Registering systemd factory
Sep 16 12:01:50.761592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.761490    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:01:50.761734 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.761723    2578 factory.go:153] Registering CRI-O factory
Sep 16 12:01:50.761795 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.761737    2578 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:01:50.761848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.761786    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:01:50.761848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.761818    2578 factory.go:103] Registering Raw factory
Sep 16 12:01:50.761848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.761834    2578 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:01:50.762172 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.762159    2578 manager.go:349] Starting recovery of all containers
Sep 16 12:01:50.764679 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.764653    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:01:50.764783 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.764754    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:01:50.773200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.773184    2578 manager.go:354] Recovery completed
Sep 16 12:01:50.778195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.778176    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sktxt"
Sep 16 12:01:50.778925 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.778912    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:50.780829 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.780812    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:50.780916 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.780849    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:50.780916 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.780864    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:50.781460 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.781445    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:01:50.781460 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.781456    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:01:50.781577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.781473    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:01:50.781577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.781501    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:50.782236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.782217    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:50.782311 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.782252    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:50.782311 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.782266    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:50.783002 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.782933    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-170.ec2.internal.18d5cb70734cd327  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-170.ec2.internal,UID:ip-10-0-136-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-170.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-170.ec2.internal,},FirstTimestamp:2026-09-16 12:01:50.780830503 +0000 UTC m=+0.418155339,LastTimestamp:2026-09-16 12:01:50.780830503 +0000 UTC m=+0.418155339,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-170.ec2.internal,}"
Sep 16 12:01:50.783673 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.783659    2578 policy_none.go:49] "None policy: Start"
Sep 16 12:01:50.783731 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.783677    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:01:50.783731 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.783687    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:01:50.785383 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.785366    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sktxt"
Sep 16 12:01:50.786632 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.786620    2578 policy_none.go:47] "Start"
Sep 16 12:01:50.793211 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.793147    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-170.ec2.internal.18d5cb70734d353d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-170.ec2.internal,UID:ip-10-0-136-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-170.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-170.ec2.internal,},FirstTimestamp:2026-09-16 12:01:50.780855613 +0000 UTC m=+0.418180442,LastTimestamp:2026-09-16 12:01:50.780855613 +0000 UTC m=+0.418180442,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-170.ec2.internal,}"
Sep 16 12:01:50.828227 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.828208    2578 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.828257    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.828271    2578 server.go:87] "Starting device plugin registration server"
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.828678    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.828692    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.828812    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.828906    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.828917    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.829441    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:01:50.840528 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.829495    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:50.876090 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.876050    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:01:50.877637 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.877622    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:01:50.877733 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.877693    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:01:50.877733 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.877725    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:01:50.877824 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.877774    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:01:50.879930 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.879900    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:50.929079 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.928971    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:50.930082 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.930064    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:50.930131 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.930105    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:50.930131 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.930117    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:50.930189 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.930147    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-170.ec2.internal"
Sep 16 12:01:50.939303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.939281    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-170.ec2.internal"
Sep 16 12:01:50.939362 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.939308    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-170.ec2.internal\": node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:50.965160 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:50.965131    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:50.977987 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.977957    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal"]
Sep 16 12:01:50.978081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.978048    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:50.979095 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.979077    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:50.979160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.979115    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:50.979160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.979131    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:50.981360 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.981348    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:50.981525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.981499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:50.981568 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.981547    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:50.982112 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.982096    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:50.982194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.982125    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:50.982194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.982125    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:50.982194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.982141    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:50.982194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.982152    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:50.982194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.982163    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:50.984487 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.984470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:50.984543 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.984507    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:50.985271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.985251    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:50.985377 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.985276    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:50.985377 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:50.985285    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.007276 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.007249    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-170.ec2.internal\" not found" node="ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.011020 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.010999    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-170.ec2.internal\" not found" node="ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.062173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.062139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/164253f176804fbadee3f3d306fd6e92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal\" (UID: \"164253f176804fbadee3f3d306fd6e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.062173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.062173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/164253f176804fbadee3f3d306fd6e92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal\" (UID: \"164253f176804fbadee3f3d306fd6e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.062301 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.062189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e474ffa0780843092bf4057d45da8093-config\") pod \"kube-apiserver-proxy-ip-10-0-136-170.ec2.internal\" (UID: \"e474ffa0780843092bf4057d45da8093\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.065201 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.065178    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.162724 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.162674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/164253f176804fbadee3f3d306fd6e92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal\" (UID: \"164253f176804fbadee3f3d306fd6e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.162828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.162734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/164253f176804fbadee3f3d306fd6e92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal\" (UID: \"164253f176804fbadee3f3d306fd6e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.162828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.162754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e474ffa0780843092bf4057d45da8093-config\") pod \"kube-apiserver-proxy-ip-10-0-136-170.ec2.internal\" (UID: \"e474ffa0780843092bf4057d45da8093\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.162828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.162799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e474ffa0780843092bf4057d45da8093-config\") pod \"kube-apiserver-proxy-ip-10-0-136-170.ec2.internal\" (UID: \"e474ffa0780843092bf4057d45da8093\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.162917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.162819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/164253f176804fbadee3f3d306fd6e92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal\" (UID: \"164253f176804fbadee3f3d306fd6e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.162917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.162849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/164253f176804fbadee3f3d306fd6e92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal\" (UID: \"164253f176804fbadee3f3d306fd6e92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.165799 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.165774    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.266744 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.266685    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.309885 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.309855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.313873 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.313854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:51.367015 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.366978    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.467535 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.467477    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.568099 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.567994    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.668867 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.668817    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.695000 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.694975    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:01:51.695405 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.695128    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:01:51.749137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.749105    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:51.769365 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.769337    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.791540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.791489    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 11:56:51 +0000 UTC" deadline="2028-04-15 12:34:28.214538794 +0000 UTC"
Sep 16 12:01:51.791691 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.791535    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13848h32m36.42300822s"
Sep 16 12:01:51.870145 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.870074    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:51.901124 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:51.901084    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164253f176804fbadee3f3d306fd6e92.slice/crio-45df2f2a1c05ff9340f220fe462fc37b076b131e0500e18bd63833d05cb3c6a5 WatchSource:0}: Error finding container 45df2f2a1c05ff9340f220fe462fc37b076b131e0500e18bd63833d05cb3c6a5: Status 404 returned error can't find the container with id 45df2f2a1c05ff9340f220fe462fc37b076b131e0500e18bd63833d05cb3c6a5
Sep 16 12:01:51.907554 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.907540    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:01:51.907787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.907773    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:51.909973 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.909944    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:51.909973 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.909976    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:51.910128 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:51.909986    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.935063 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:51.935033    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode474ffa0780843092bf4057d45da8093.slice/crio-76c57adf1ca991ead84a9cc1531cbfef152b5497f5e3325613967977813a6ed0 WatchSource:0}: Error finding container 76c57adf1ca991ead84a9cc1531cbfef152b5497f5e3325613967977813a6ed0: Status 404 returned error can't find the container with id 76c57adf1ca991ead84a9cc1531cbfef152b5497f5e3325613967977813a6ed0
Sep 16 12:01:51.971150 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:51.971117    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:52.071726 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.071688    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:52.170917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.170887    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:52.171839 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.171819    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-170.ec2.internal\" not found"
Sep 16 12:01:52.199280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.199257    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:52.260656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.260620    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:52.274404 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.274382    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:01:52.275516 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.275501    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal"
Sep 16 12:01:52.282661 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.282644    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:01:52.736294 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.736239    2578 apiserver.go:52] "Watching apiserver"
Sep 16 12:01:52.746221 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.746196    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:01:52.747391 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.747366    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-wt7g7","openshift-multus/multus-52lz7","openshift-network-operator/iptables-alerter-wh9cp","openshift-ovn-kubernetes/ovnkube-node-kr5vm","kube-system/konnectivity-agent-hcxdk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q","openshift-cluster-node-tuning-operator/tuned-sbcwv","openshift-dns/node-resolver-kw862","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal","openshift-multus/multus-additional-cni-plugins-2xqd9","openshift-multus/network-metrics-daemon-nhjbr","openshift-network-diagnostics/network-check-target-j7tg4","kube-system/global-pull-secret-syncer-8fst2","kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal"]
Sep 16 12:01:52.750356 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.750333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.752560 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.752536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.752694 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.752609    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:01:52.753157 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.753124    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:01:52.753269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.753195    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:01:52.753337 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.753311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-444nq\""
Sep 16 12:01:52.753424 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.753401    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:01:52.753551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.753411    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:01:52.754036 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.754017    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:01:52.754966 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.754946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.757496 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.757192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.759391 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.759377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:01:52.759928 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.759914    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:01:52.761742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.761726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.764031 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.764016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:52.764116 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.764094    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:01:52.765348 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.765318    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:01:52.765348 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.765323    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:01:52.765451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.765323    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:01:52.765611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.765596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:01:52.765656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.765624    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:01:52.765752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.765740    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:01:52.766436 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.766423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.766819 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.766783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wkfhq\""
Sep 16 12:01:52.766819 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.766804    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kk577\""
Sep 16 12:01:52.766975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.766877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:01:52.766975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.766882    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:01:52.767063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.767002    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tf62w\""
Sep 16 12:01:52.767114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.767086    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:01:52.767167 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.767131    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:01:52.767167 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.767153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vmkrn\""
Sep 16 12:01:52.769209 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.769191    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:01:52.769345 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.769325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.769444 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.769403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:01:52.769501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.769465    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xsmws\""
Sep 16 12:01:52.771912 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.771879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.772818 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.772775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.772934 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.772825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c58447-61b8-400e-be4b-8519523f5788-kubelet-config\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.772934 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.772858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hg2\" (UniqueName: \"kubernetes.io/projected/a0480890-6eaf-4f5f-bd35-3d19ee846601-kube-api-access-v8hg2\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.773046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.772891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.773046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.772980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.773046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-sys-fs\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.773200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-cnibin\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.773200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773179    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:01:52.773303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.773303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773295    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tdx8w\""
Sep 16 12:01:52.773403 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c58447-61b8-400e-be4b-8519523f5788-dbus\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.773403 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.773403 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0480890-6eaf-4f5f-bd35-3d19ee846601-host-slash\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.773555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-device-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.773555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9tb\" (UniqueName: \"kubernetes.io/projected/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-kube-api-access-dl9tb\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.773555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.773555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zmn\" (UniqueName: \"kubernetes.io/projected/b2238633-58be-4323-940e-5aa992160526-kube-api-access-b4zmn\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.773804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0dfca71f-6cb4-432e-9a94-339724df3d63-konnectivity-ca\") pod \"konnectivity-agent-hcxdk\" (UID: \"0dfca71f-6cb4-432e-9a94-339724df3d63\") " pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:01:52.773804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-hosts-file\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.773804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-tmp-dir\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.773804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0dfca71f-6cb4-432e-9a94-339724df3d63-agent-certs\") pod \"konnectivity-agent-hcxdk\" (UID: \"0dfca71f-6cb4-432e-9a94-339724df3d63\") " pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:01:52.773804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.773804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0480890-6eaf-4f5f-bd35-3d19ee846601-iptables-alerter-script\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.774103 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjkb\" (UniqueName: \"kubernetes.io/projected/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-kube-api-access-lrjkb\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.774103 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-socket-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.774103 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-registration-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.774103 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.773970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-system-cni-dir\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.775242 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.775217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:01:52.775351 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.775305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:52.775422 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.775373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:01:52.775422 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.775384    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:01:52.775527 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.775478    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2n2n7\""
Sep 16 12:01:52.775686 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.775671    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:01:52.777765 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.777748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.779124 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.779104    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:01:52.780212 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.780194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rbr8r\""
Sep 16 12:01:52.780306 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.780269    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:01:52.780373 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.780324    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:01:52.780615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.780441    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:01:52.780704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.780625    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:01:52.781725 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.781706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:01:52.781816 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.781805    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:01:52.814190 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.814169    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-shbgb"
Sep 16 12:01:52.825599 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.825573    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-shbgb"
Sep 16 12:01:52.861415 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.861377    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:01:52.874175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0dfca71f-6cb4-432e-9a94-339724df3d63-konnectivity-ca\") pod \"konnectivity-agent-hcxdk\" (UID: \"0dfca71f-6cb4-432e-9a94-339724df3d63\") " pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:01:52.874281 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-run-netns\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.874281 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-lib-modules\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.874281 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-host\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.874387 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-netns\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.874387 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-cni-bin\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.874449 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-socket-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.874489 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-kubelet\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.874525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-run-ovn-kubernetes\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.874578 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-ovnkube-config\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.874628 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfmt\" (UniqueName: \"kubernetes.io/projected/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-kube-api-access-qxfmt\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.874628 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-sys-fs\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.874628 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-socket-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.874764 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.874764 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-etc-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.874764 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-sys-fs\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.874764 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-run\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.874971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0480890-6eaf-4f5f-bd35-3d19ee846601-host-slash\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.874971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-device-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.874971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0480890-6eaf-4f5f-bd35-3d19ee846601-host-slash\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.874971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0dfca71f-6cb4-432e-9a94-339724df3d63-konnectivity-ca\") pod \"konnectivity-agent-hcxdk\" (UID: \"0dfca71f-6cb4-432e-9a94-339724df3d63\") " pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:01:52.874971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-device-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.874971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysctl-d\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.874971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-hosts-file\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.874971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0dfca71f-6cb4-432e-9a94-339724df3d63-agent-certs\") pod \"konnectivity-agent-hcxdk\" (UID: \"0dfca71f-6cb4-432e-9a94-339724df3d63\") " pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.874987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-hosts-file\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-system-cni-dir\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-log-socket\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-system-cni-dir\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-socket-dir-parent\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ntb\" (UniqueName: \"kubernetes.io/projected/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-kube-api-access-z2ntb\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c58447-61b8-400e-be4b-8519523f5788-kubelet-config\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hg2\" (UniqueName: \"kubernetes.io/projected/a0480890-6eaf-4f5f-bd35-3d19ee846601-kube-api-access-v8hg2\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c58447-61b8-400e-be4b-8519523f5788-kubelet-config\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.875355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-cnibin\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-var-lib-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8j7\" (UniqueName: \"kubernetes.io/projected/2112a93a-a830-4c7b-a86e-657507d2d927-kube-api-access-mb8j7\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2238633-58be-4323-940e-5aa992160526-cnibin\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-tmp\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c58447-61b8-400e-be4b-8519523f5788-dbus\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9tb\" (UniqueName: \"kubernetes.io/projected/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-kube-api-access-dl9tb\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-node-log\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2112a93a-a830-4c7b-a86e-657507d2d927-ovn-node-metrics-cert\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5cd30a3-5801-4d66-97f2-7361a03672d8-cni-binary-copy\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-k8s-cni-cncf-io\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.876168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c58447-61b8-400e-be4b-8519523f5788-dbus\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-daemon-config\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-tmp-dir\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-slash\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875866    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-systemd\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-sys\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.875972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-cnibin\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-kubelet\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2238633-58be-4323-940e-5aa992160526-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-host\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-tmp-dir\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-cni-bin\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-ovnkube-script-lib\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysctl-conf\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-multus-certs\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58tv\" (UniqueName: \"kubernetes.io/projected/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-kube-api-access-h58tv\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:52.876979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-kubernetes\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-var-lib-kubelet\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-conf-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-cni-netd\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.876579    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.876660    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret podName:04c58447-61b8-400e-be4b-8519523f5788 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:53.376630913 +0000 UTC m=+3.013955758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret") pod "global-pull-secret-syncer-8fst2" (UID: "04c58447-61b8-400e-be4b-8519523f5788") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-env-overrides\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-cni-multus\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-etc-kubernetes\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-ovn\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-systemd\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-tuned\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-cni-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-hostroot\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ftzs\" (UniqueName: \"kubernetes.io/projected/f5cd30a3-5801-4d66-97f2-7361a03672d8-kube-api-access-8ftzs\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.877692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.876985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-serviceca\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0480890-6eaf-4f5f-bd35-3d19ee846601-iptables-alerter-script\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjkb\" (UniqueName: \"kubernetes.io/projected/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-kube-api-access-lrjkb\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-registration-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-systemd-units\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-modprobe-d\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysconfig\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-registration-dir\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zmn\" (UniqueName: \"kubernetes.io/projected/b2238633-58be-4323-940e-5aa992160526-kube-api-access-b4zmn\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-system-cni-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.878251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.877601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0480890-6eaf-4f5f-bd35-3d19ee846601-iptables-alerter-script\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.880615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.880591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0dfca71f-6cb4-432e-9a94-339724df3d63-agent-certs\") pod \"konnectivity-agent-hcxdk\" (UID: \"0dfca71f-6cb4-432e-9a94-339724df3d63\") " pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:01:52.882816 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.882771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal" event={"ID":"e474ffa0780843092bf4057d45da8093","Type":"ContainerStarted","Data":"76c57adf1ca991ead84a9cc1531cbfef152b5497f5e3325613967977813a6ed0"}
Sep 16 12:01:52.884080 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.884054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal" event={"ID":"164253f176804fbadee3f3d306fd6e92","Type":"ContainerStarted","Data":"45df2f2a1c05ff9340f220fe462fc37b076b131e0500e18bd63833d05cb3c6a5"}
Sep 16 12:01:52.884418 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.884402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hg2\" (UniqueName: \"kubernetes.io/projected/a0480890-6eaf-4f5f-bd35-3d19ee846601-kube-api-access-v8hg2\") pod \"iptables-alerter-wh9cp\" (UID: \"a0480890-6eaf-4f5f-bd35-3d19ee846601\") " pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:52.888100 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.888071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zmn\" (UniqueName: \"kubernetes.io/projected/b2238633-58be-4323-940e-5aa992160526-kube-api-access-b4zmn\") pod \"multus-additional-cni-plugins-2xqd9\" (UID: \"b2238633-58be-4323-940e-5aa992160526\") " pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:52.888100 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.888067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjkb\" (UniqueName: \"kubernetes.io/projected/4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f-kube-api-access-lrjkb\") pod \"node-resolver-kw862\" (UID: \"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f\") " pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:52.888726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.888705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9tb\" (UniqueName: \"kubernetes.io/projected/3355b66b-9d8b-4d6c-a316-fe38ff6b49ec-kube-api-access-dl9tb\") pod \"aws-ebs-csi-driver-node-q584q\" (UID: \"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:52.978249 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-tmp\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.978438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-node-log\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2112a93a-a830-4c7b-a86e-657507d2d927-ovn-node-metrics-cert\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5cd30a3-5801-4d66-97f2-7361a03672d8-cni-binary-copy\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.978438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-k8s-cni-cncf-io\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.978438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-daemon-config\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.978438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-slash\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-node-log\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-k8s-cni-cncf-io\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-systemd\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-systemd\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-sys\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-sys\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-slash\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-cnibin\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-kubelet\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-host\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-cni-bin\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-ovnkube-script-lib\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.978787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-kubelet\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-cnibin\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-cni-bin\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysctl-conf\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-host\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-multus-certs\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58tv\" (UniqueName: \"kubernetes.io/projected/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-kube-api-access-h58tv\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-kubernetes\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-daemon-config\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.978976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-var-lib-kubelet\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-var-lib-kubelet\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-conf-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysctl-conf\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-multus-certs\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-conf-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5cd30a3-5801-4d66-97f2-7361a03672d8-cni-binary-copy\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.979367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-cni-netd\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-kubernetes\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-env-overrides\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-cni-netd\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-cni-multus\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-etc-kubernetes\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-ovn\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-cni-multus\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-systemd\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-etc-kubernetes\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-tuned\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-ovn\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-systemd\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-cni-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-hostroot\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ftzs\" (UniqueName: \"kubernetes.io/projected/f5cd30a3-5801-4d66-97f2-7361a03672d8-kube-api-access-8ftzs\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-serviceca\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-hostroot\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.980309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-cni-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-systemd-units\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-modprobe-d\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysconfig\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-ovnkube-script-lib\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-system-cni-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-run-netns\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-lib-modules\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-env-overrides\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-host\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-netns\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-modprobe-d\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-cni-bin\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-run-netns\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-system-cni-dir\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-kubelet\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-var-lib-cni-bin\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.981027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-run-ovn-kubernetes\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysconfig\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-ovnkube-config\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfmt\" (UniqueName: \"kubernetes.io/projected/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-kube-api-access-qxfmt\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-systemd-units\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-etc-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-run\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-lib-modules\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysctl-d\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.979999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-host\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-host-run-netns\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-etc-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-kubelet\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-log-socket\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-socket-dir-parent\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ntb\" (UniqueName: \"kubernetes.io/projected/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-kube-api-access-z2ntb\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.981559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-sysctl-d\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-var-lib-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8j7\" (UniqueName: \"kubernetes.io/projected/2112a93a-a830-4c7b-a86e-657507d2d927-kube-api-access-mb8j7\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-var-lib-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-run-openvswitch\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-serviceca\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.980338    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.980403    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs podName:16ec4baa-3a59-4ba5-afd3-eafb08071c5a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:53.480380811 +0000 UTC m=+3.117705646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs") pod "network-metrics-daemon-nhjbr" (UID: "16ec4baa-3a59-4ba5-afd3-eafb08071c5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-host-run-ovn-kubernetes\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-run\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5cd30a3-5801-4d66-97f2-7361a03672d8-multus-socket-dir-parent\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.980653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2112a93a-a830-4c7b-a86e-657507d2d927-log-socket\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.981317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2112a93a-a830-4c7b-a86e-657507d2d927-ovn-node-metrics-cert\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.981357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2112a93a-a830-4c7b-a86e-657507d2d927-ovnkube-config\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.981499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-tmp\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.982111 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.981725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-etc-tuned\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:52.986748 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.986688    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:52.986748 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.986712    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:52.986748 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.986728    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v7ntk for pod openshift-network-diagnostics/network-check-target-j7tg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:52.987060 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:52.986804    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk podName:7a847c41-95f4-40fb-aaed-ee5f5057584d nodeName:}" failed. No retries permitted until 2026-09-16 12:01:53.486784182 +0000 UTC m=+3.124109022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v7ntk" (UniqueName: "kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk") pod "network-check-target-j7tg4" (UID: "7a847c41-95f4-40fb-aaed-ee5f5057584d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:52.989355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.989316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ftzs\" (UniqueName: \"kubernetes.io/projected/f5cd30a3-5801-4d66-97f2-7361a03672d8-kube-api-access-8ftzs\") pod \"multus-52lz7\" (UID: \"f5cd30a3-5801-4d66-97f2-7361a03672d8\") " pod="openshift-multus/multus-52lz7"
Sep 16 12:01:52.989619 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.989596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58tv\" (UniqueName: \"kubernetes.io/projected/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-kube-api-access-h58tv\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:52.990836 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.990814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8j7\" (UniqueName: \"kubernetes.io/projected/2112a93a-a830-4c7b-a86e-657507d2d927-kube-api-access-mb8j7\") pod \"ovnkube-node-kr5vm\" (UID: \"2112a93a-a830-4c7b-a86e-657507d2d927\") " pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:52.990926 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.990842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ntb\" (UniqueName: \"kubernetes.io/projected/ef67b5a6-8ba4-41b0-9f63-3a242a942d3c-kube-api-access-z2ntb\") pod \"node-ca-wt7g7\" (UID: \"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c\") " pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:52.990969 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:52.990931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfmt\" (UniqueName: \"kubernetes.io/projected/544c5ff2-582d-4fc0-bec7-1efbf1f34dbe-kube-api-access-qxfmt\") pod \"tuned-sbcwv\" (UID: \"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:53.063648 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.063611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2xqd9"
Sep 16 12:01:53.067760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.067737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wh9cp"
Sep 16 12:01:53.079022 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.078833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kw862"
Sep 16 12:01:53.084235 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.084210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:01:53.089304 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.089281    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q"
Sep 16 12:01:53.090867 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:53.090792    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbddbd3_dda8_46a7_b3ac_a4d8fada3f1f.slice/crio-eb06b0017f809f033366da79ff5ac69348e060f28ff28340187ad7018f3043f2 WatchSource:0}: Error finding container eb06b0017f809f033366da79ff5ac69348e060f28ff28340187ad7018f3043f2: Status 404 returned error can't find the container with id eb06b0017f809f033366da79ff5ac69348e060f28ff28340187ad7018f3043f2
Sep 16 12:01:53.095364 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:53.095336    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfca71f_6cb4_432e_9a94_339724df3d63.slice/crio-acb63b512cbda93aa7dac3e15b5731a3341e0e2794112a48afbfafd19521301a WatchSource:0}: Error finding container acb63b512cbda93aa7dac3e15b5731a3341e0e2794112a48afbfafd19521301a: Status 404 returned error can't find the container with id acb63b512cbda93aa7dac3e15b5731a3341e0e2794112a48afbfafd19521301a
Sep 16 12:01:53.096154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.096128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sbcwv"
Sep 16 12:01:53.100934 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:53.100911    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3355b66b_9d8b_4d6c_a316_fe38ff6b49ec.slice/crio-dfdfa2bb6fb82ccfe4eb7add0c31a17caf89c5226c47e0744ce9abb75fad02d4 WatchSource:0}: Error finding container dfdfa2bb6fb82ccfe4eb7add0c31a17caf89c5226c47e0744ce9abb75fad02d4: Status 404 returned error can't find the container with id dfdfa2bb6fb82ccfe4eb7add0c31a17caf89c5226c47e0744ce9abb75fad02d4
Sep 16 12:01:53.101059 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.100976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-52lz7"
Sep 16 12:01:53.106114 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:53.106091    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544c5ff2_582d_4fc0_bec7_1efbf1f34dbe.slice/crio-826203bfe004731afc459fff7c7128c1d2a8ae4632ca6251ca95ed4176ef7370 WatchSource:0}: Error finding container 826203bfe004731afc459fff7c7128c1d2a8ae4632ca6251ca95ed4176ef7370: Status 404 returned error can't find the container with id 826203bfe004731afc459fff7c7128c1d2a8ae4632ca6251ca95ed4176ef7370
Sep 16 12:01:53.107468 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.107447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wt7g7"
Sep 16 12:01:53.111676 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:53.111649    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cd30a3_5801_4d66_97f2_7361a03672d8.slice/crio-ff5bcb052820bab009bad01bb5fd5137c13dcf836b8ac9a093a1b31d407feaff WatchSource:0}: Error finding container ff5bcb052820bab009bad01bb5fd5137c13dcf836b8ac9a093a1b31d407feaff: Status 404 returned error can't find the container with id ff5bcb052820bab009bad01bb5fd5137c13dcf836b8ac9a093a1b31d407feaff
Sep 16 12:01:53.112374 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.112348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:01:53.117474 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:53.117448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef67b5a6_8ba4_41b0_9f63_3a242a942d3c.slice/crio-2fa9b5b5e617b0a84e7eb727053c58984b142930e960c974ade349eec4a26b41 WatchSource:0}: Error finding container 2fa9b5b5e617b0a84e7eb727053c58984b142930e960c974ade349eec4a26b41: Status 404 returned error can't find the container with id 2fa9b5b5e617b0a84e7eb727053c58984b142930e960c974ade349eec4a26b41
Sep 16 12:01:53.121905 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:01:53.121885    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2112a93a_a830_4c7b_a86e_657507d2d927.slice/crio-752cf4d978e54d6d933579d4d8ddcbef9b309c023fb7e71b16f89f165a92b55c WatchSource:0}: Error finding container 752cf4d978e54d6d933579d4d8ddcbef9b309c023fb7e71b16f89f165a92b55c: Status 404 returned error can't find the container with id 752cf4d978e54d6d933579d4d8ddcbef9b309c023fb7e71b16f89f165a92b55c
Sep 16 12:01:53.259579 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.259464    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:53.384080 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.384029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:53.384283 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:53.384253    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:53.384353 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:53.384338    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret podName:04c58447-61b8-400e-be4b-8519523f5788 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:54.384315781 +0000 UTC m=+4.021640606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret") pod "global-pull-secret-syncer-8fst2" (UID: "04c58447-61b8-400e-be4b-8519523f5788") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:53.485234 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.485191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:53.485445 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:53.485333    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:53.485445 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:53.485422    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs podName:16ec4baa-3a59-4ba5-afd3-eafb08071c5a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:54.485401507 +0000 UTC m=+4.122726343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs") pod "network-metrics-daemon-nhjbr" (UID: "16ec4baa-3a59-4ba5-afd3-eafb08071c5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:53.586064 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.585973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:53.586225 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:53.586142    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:53.586225 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:53.586159    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:53.586225 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:53.586168    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v7ntk for pod openshift-network-diagnostics/network-check-target-j7tg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:53.586344 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:53.586231    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk podName:7a847c41-95f4-40fb-aaed-ee5f5057584d nodeName:}" failed. No retries permitted until 2026-09-16 12:01:54.586212126 +0000 UTC m=+4.223536967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7ntk" (UniqueName: "kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk") pod "network-check-target-j7tg4" (UID: "7a847c41-95f4-40fb-aaed-ee5f5057584d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:53.829638 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.829438    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:56:53 +0000 UTC" deadline="2028-03-06 23:48:46.432655455 +0000 UTC"
Sep 16 12:01:53.829638 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.829638    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12899h46m52.603023624s"
Sep 16 12:01:53.890103 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.889991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wh9cp" event={"ID":"a0480890-6eaf-4f5f-bd35-3d19ee846601","Type":"ContainerStarted","Data":"d7ea6818df2fd1939afb5be104e96e336bec8b2ebacf5673b4db6127aabb9a1f"}
Sep 16 12:01:53.893823 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.893786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wt7g7" event={"ID":"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c","Type":"ContainerStarted","Data":"2fa9b5b5e617b0a84e7eb727053c58984b142930e960c974ade349eec4a26b41"}
Sep 16 12:01:53.896405 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.896375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-52lz7" event={"ID":"f5cd30a3-5801-4d66-97f2-7361a03672d8","Type":"ContainerStarted","Data":"ff5bcb052820bab009bad01bb5fd5137c13dcf836b8ac9a093a1b31d407feaff"}
Sep 16 12:01:53.898001 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.897974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sbcwv" event={"ID":"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe","Type":"ContainerStarted","Data":"826203bfe004731afc459fff7c7128c1d2a8ae4632ca6251ca95ed4176ef7370"}
Sep 16 12:01:53.901247 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.901199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q" event={"ID":"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec","Type":"ContainerStarted","Data":"dfdfa2bb6fb82ccfe4eb7add0c31a17caf89c5226c47e0744ce9abb75fad02d4"}
Sep 16 12:01:53.913994 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.913911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hcxdk" event={"ID":"0dfca71f-6cb4-432e-9a94-339724df3d63","Type":"ContainerStarted","Data":"acb63b512cbda93aa7dac3e15b5731a3341e0e2794112a48afbfafd19521301a"}
Sep 16 12:01:53.915857 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.915786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kw862" event={"ID":"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f","Type":"ContainerStarted","Data":"eb06b0017f809f033366da79ff5ac69348e060f28ff28340187ad7018f3043f2"}
Sep 16 12:01:53.918057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.917990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" event={"ID":"b2238633-58be-4323-940e-5aa992160526","Type":"ContainerStarted","Data":"091401a316ba45550b67164bf06df970d4bcb1336c3fb40dcaaa7a57943e776a"}
Sep 16 12:01:53.920472 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:53.920445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"752cf4d978e54d6d933579d4d8ddcbef9b309c023fb7e71b16f89f165a92b55c"}
Sep 16 12:01:54.393701 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:54.393660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:54.393898 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.393879    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:54.393964 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.393951    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret podName:04c58447-61b8-400e-be4b-8519523f5788 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:56.393930559 +0000 UTC m=+6.031255380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret") pod "global-pull-secret-syncer-8fst2" (UID: "04c58447-61b8-400e-be4b-8519523f5788") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:54.494294 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:54.494258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:54.494496 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.494441    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:54.494652 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.494581    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs podName:16ec4baa-3a59-4ba5-afd3-eafb08071c5a nodeName:}" failed. No retries permitted until 2026-09-16 12:01:56.494541084 +0000 UTC m=+6.131865903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs") pod "network-metrics-daemon-nhjbr" (UID: "16ec4baa-3a59-4ba5-afd3-eafb08071c5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:54.595569 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:54.595461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:54.595737 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.595674    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:54.595737 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.595702    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:54.595737 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.595717    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v7ntk for pod openshift-network-diagnostics/network-check-target-j7tg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:54.595907 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.595791    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk podName:7a847c41-95f4-40fb-aaed-ee5f5057584d nodeName:}" failed. No retries permitted until 2026-09-16 12:01:56.595770716 +0000 UTC m=+6.233095537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7ntk" (UniqueName: "kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk") pod "network-check-target-j7tg4" (UID: "7a847c41-95f4-40fb-aaed-ee5f5057584d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:54.831297 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:54.831250    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:56:53 +0000 UTC" deadline="2028-06-09 16:02:28.936346284 +0000 UTC"
Sep 16 12:01:54.831297 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:54.831295    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15172h0m34.105055575s"
Sep 16 12:01:54.884443 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:54.884349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:54.884627 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.884479    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:01:54.885005 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:54.884982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:54.885140 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.885091    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:01:54.885213 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:54.885161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:54.885270 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:54.885241    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:01:55.952697 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:55.951838    2578 generic.go:356] "Generic (PLEG): container finished" podID="164253f176804fbadee3f3d306fd6e92" containerID="defe763061dd2f99c880d5f32f451f5d15b696477ebecff6c49090787721a8a1" exitCode=0
Sep 16 12:01:55.952697 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:55.951900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal" event={"ID":"164253f176804fbadee3f3d306fd6e92","Type":"ContainerDied","Data":"defe763061dd2f99c880d5f32f451f5d15b696477ebecff6c49090787721a8a1"}
Sep 16 12:01:56.414989 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:56.414889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:56.415175 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.415117    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:56.415248 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.415193    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret podName:04c58447-61b8-400e-be4b-8519523f5788 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:00.415166105 +0000 UTC m=+10.052490938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret") pod "global-pull-secret-syncer-8fst2" (UID: "04c58447-61b8-400e-be4b-8519523f5788") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:56.516304 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:56.516262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:56.516500 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.516432    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:56.516572 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.516505    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs podName:16ec4baa-3a59-4ba5-afd3-eafb08071c5a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:00.516483737 +0000 UTC m=+10.153808580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs") pod "network-metrics-daemon-nhjbr" (UID: "16ec4baa-3a59-4ba5-afd3-eafb08071c5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:56.617375 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:56.617333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:56.617612 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.617583    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:56.617612 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.617613    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:56.617766 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.617627    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v7ntk for pod openshift-network-diagnostics/network-check-target-j7tg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:56.617766 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.617700    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk podName:7a847c41-95f4-40fb-aaed-ee5f5057584d nodeName:}" failed. No retries permitted until 2026-09-16 12:02:00.617681024 +0000 UTC m=+10.255005847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7ntk" (UniqueName: "kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk") pod "network-check-target-j7tg4" (UID: "7a847c41-95f4-40fb-aaed-ee5f5057584d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:56.878733 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:56.878649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:56.878905 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.878810    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:01:56.879237 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:56.879210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:56.879339 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.879321    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:01:56.881714 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:56.881688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:56.881836 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:56.881810    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:01:58.879091 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:58.879051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:01:58.879606 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:58.879052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:01:58.879606 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:58.879190    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:01:58.879606 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:01:58.879210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:01:58.879606 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:58.879297    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:01:58.879606 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:01:58.879395    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:00.456789 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:00.456742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:00.457361 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.456924    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:00.457361 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.456987    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret podName:04c58447-61b8-400e-be4b-8519523f5788 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:08.456967596 +0000 UTC m=+18.094292418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret") pod "global-pull-secret-syncer-8fst2" (UID: "04c58447-61b8-400e-be4b-8519523f5788") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:00.558259 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:00.557659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:00.558259 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.557847    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:00.558259 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.557917    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs podName:16ec4baa-3a59-4ba5-afd3-eafb08071c5a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:08.557893791 +0000 UTC m=+18.195218624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs") pod "network-metrics-daemon-nhjbr" (UID: "16ec4baa-3a59-4ba5-afd3-eafb08071c5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:00.658509 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:00.658472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:00.658730 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.658663    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:02:00.658730 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.658686    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:02:00.658730 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.658696    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v7ntk for pod openshift-network-diagnostics/network-check-target-j7tg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:00.658913 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.658765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk podName:7a847c41-95f4-40fb-aaed-ee5f5057584d nodeName:}" failed. No retries permitted until 2026-09-16 12:02:08.658743563 +0000 UTC m=+18.296068401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7ntk" (UniqueName: "kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk") pod "network-check-target-j7tg4" (UID: "7a847c41-95f4-40fb-aaed-ee5f5057584d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:00.878653 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:00.878569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:00.878791 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.878678    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:00.879037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:00.879017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:00.879188 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.879164    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:00.879324 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:00.879214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:00.879324 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:00.879279    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:02.878212 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:02.878170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:02.878623 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:02.878307    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:02.878623 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:02.878185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:02.878623 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:02.878328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:02.878623 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:02.878436    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:02.878623 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:02.878588    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:04.878534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:04.878387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:04.878534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:04.878424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:04.878534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:04.878392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:04.879056 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:04.878633    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:04.879056 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:04.878740    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:04.879192 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:04.879154    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:06.878227 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:06.878191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:06.878639 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:06.878199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:06.878639 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:06.878343    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:06.878639 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:06.878366    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:06.878639 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:06.878472    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:06.878639 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:06.878583    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:08.511038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:08.511004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:08.511567 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.511167    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:08.511567 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.511246    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret podName:04c58447-61b8-400e-be4b-8519523f5788 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:24.511225543 +0000 UTC m=+34.148550376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret") pod "global-pull-secret-syncer-8fst2" (UID: "04c58447-61b8-400e-be4b-8519523f5788") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:08.612141 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:08.612112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:08.612321 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.612272    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:08.612378 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.612342    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs podName:16ec4baa-3a59-4ba5-afd3-eafb08071c5a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:24.612326649 +0000 UTC m=+34.249651466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs") pod "network-metrics-daemon-nhjbr" (UID: "16ec4baa-3a59-4ba5-afd3-eafb08071c5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:08.713257 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:08.713173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:08.713441 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.713312    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:02:08.713441 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.713333    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:02:08.713441 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.713347    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v7ntk for pod openshift-network-diagnostics/network-check-target-j7tg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:08.713441 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.713406    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk podName:7a847c41-95f4-40fb-aaed-ee5f5057584d nodeName:}" failed. No retries permitted until 2026-09-16 12:02:24.713392514 +0000 UTC m=+34.350717330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7ntk" (UniqueName: "kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk") pod "network-check-target-j7tg4" (UID: "7a847c41-95f4-40fb-aaed-ee5f5057584d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:08.878262 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:08.878176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:08.878262 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:08.878209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:08.878262 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:08.878176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:08.878558 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.878316    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:08.878558 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.878393    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:08.878558 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:08.878478    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:10.878977 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:10.878942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:10.879434 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:10.879044    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:10.879434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:10.879100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:10.879434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:10.879162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:10.879434 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:10.879242    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:10.879434 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:10.879302    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:12.878638 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:12.878600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:12.879093 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:12.878601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:12.879093 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:12.878737    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:12.879093 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:12.878608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:12.879093 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:12.878853    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:12.879093 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:12.878971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:13.997871 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:13.997395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal" event={"ID":"e474ffa0780843092bf4057d45da8093","Type":"ContainerStarted","Data":"05ddd5824e8a006279182ca6bf4f9be408a009e517130039a3c6281c9d998780"}
Sep 16 12:02:13.998956 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:13.998921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wt7g7" event={"ID":"ef67b5a6-8ba4-41b0-9f63-3a242a942d3c","Type":"ContainerStarted","Data":"acc4a5c4ec6a7900d6682bf8060840766c3f3473cb43e0a09c4cad51fa7a4167"}
Sep 16 12:02:14.000384 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.000361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-52lz7" event={"ID":"f5cd30a3-5801-4d66-97f2-7361a03672d8","Type":"ContainerStarted","Data":"eb4bf181241d90d64a45f6e3916553dd6d4469dc6cc0b7a25420337a4e582945"}
Sep 16 12:02:14.001916 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.001886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sbcwv" event={"ID":"544c5ff2-582d-4fc0-bec7-1efbf1f34dbe","Type":"ContainerStarted","Data":"132540e63a32d12314b429ee13fe8fc55abb123f7ddcf6f987a9132f7f8ca457"}
Sep 16 12:02:14.003261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.003242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q" event={"ID":"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec","Type":"ContainerStarted","Data":"660faa2243b1b806c254f714fafd6fa122a9d5495020e01451860c1d5de72406"}
Sep 16 12:02:14.004682 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.004660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hcxdk" event={"ID":"0dfca71f-6cb4-432e-9a94-339724df3d63","Type":"ContainerStarted","Data":"0798fa03b3aa0f56e4d404e66395280fb96b215fdc224c7f87c24e8a073f11ed"}
Sep 16 12:02:14.006088 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.006064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kw862" event={"ID":"4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f","Type":"ContainerStarted","Data":"04635cb2d810e21fe045789c742857563f89e48f58b7aadd90e9732bf33a5b4b"}
Sep 16 12:02:14.007573 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.007553    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2238633-58be-4323-940e-5aa992160526" containerID="c4880283f9d0b7726c957d9269722a9a80cde9ae1ff5428ae6fd27b54fa513df" exitCode=0
Sep 16 12:02:14.007674 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.007615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" event={"ID":"b2238633-58be-4323-940e-5aa992160526","Type":"ContainerDied","Data":"c4880283f9d0b7726c957d9269722a9a80cde9ae1ff5428ae6fd27b54fa513df"}
Sep 16 12:02:14.009503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.009484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal" event={"ID":"164253f176804fbadee3f3d306fd6e92","Type":"ContainerStarted","Data":"4e7e9044476b3f57b71425abd6e518733cb8107f10be43ee7a565c0155df9c8a"}
Sep 16 12:02:14.010572 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.010508    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-170.ec2.internal" podStartSLOduration=22.010493749 podStartE2EDuration="22.010493749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.010212741 +0000 UTC m=+23.647537578" watchObservedRunningTime="2026-09-16 12:02:14.010493749 +0000 UTC m=+23.647818590"
Sep 16 12:02:14.012799 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.012781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"41217bf536e47ee2e8dc8e315265a2040468b23a9b2ec0f05f1ddabc139ef866"}
Sep 16 12:02:14.012883 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.012836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"595702b92d5df7601b75fc093bdb3670cf5cb57cb56576885ddd1a7edb6c49cb"}
Sep 16 12:02:14.012883 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.012873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"3978118858c8d93bf41a0cd1ba4ff7bcedb89aedfcdd5f4a3cf84ba305eb50bd"}
Sep 16 12:02:14.012953 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.012900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"8ec6212d701532396e92df4df4c4f978c35b0f3319ac395e8a5db314c4592a27"}
Sep 16 12:02:14.012953 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.012934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"18bc771d51dc618c39bc0673ea973d168587bb66b9e9ad289f29a455cb5ca1d3"}
Sep 16 12:02:14.013021 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.012967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"69ba4a82401d16bd885602276e0a4d459821f4e4d91aa9904d2767408a9b4f48"}
Sep 16 12:02:14.043526 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.043464    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kw862" podStartSLOduration=3.143746134 podStartE2EDuration="23.043432142s" totalImagesPullingTime="19.899686008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.043243721 +0000 UTC m=+23.680568562" watchObservedRunningTime="2026-09-16 12:02:14.043432142 +0000 UTC m=+23.680756979"
Sep 16 12:02:14.059428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.059385    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sbcwv" podStartSLOduration=3.173292302 podStartE2EDuration="23.059370788s" totalImagesPullingTime="19.886078486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.058846384 +0000 UTC m=+23.696171222" watchObservedRunningTime="2026-09-16 12:02:14.059370788 +0000 UTC m=+23.696695626"
Sep 16 12:02:14.073024 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.072959    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-170.ec2.internal" podStartSLOduration=22.072939702 podStartE2EDuration="22.072939702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.072070346 +0000 UTC m=+23.709395183" watchObservedRunningTime="2026-09-16 12:02:14.072939702 +0000 UTC m=+23.710264542"
Sep 16 12:02:14.116952 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.116906    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-52lz7" podStartSLOduration=3.16667123 podStartE2EDuration="23.116893523s" totalImagesPullingTime="19.950222293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.116718101 +0000 UTC m=+23.754042940" watchObservedRunningTime="2026-09-16 12:02:14.116893523 +0000 UTC m=+23.754218362"
Sep 16 12:02:14.117142 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.117123    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hcxdk" podStartSLOduration=3.220123663 podStartE2EDuration="23.117118298s" totalImagesPullingTime="19.896994635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.096175296 +0000 UTC m=+23.733500130" watchObservedRunningTime="2026-09-16 12:02:14.117118298 +0000 UTC m=+23.754443135"
Sep 16 12:02:14.143614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.143565    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wt7g7" podStartSLOduration=6.971939051 podStartE2EDuration="23.143545943s" totalImagesPullingTime="16.171606892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.143185462 +0000 UTC m=+23.780510300" watchObservedRunningTime="2026-09-16 12:02:14.143545943 +0000 UTC m=+23.780870781"
Sep 16 12:02:14.676993 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.676960    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:02:14.844409 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.844254    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:02:14.676987341Z","UUID":"d6df12f6-d1ee-4405-9edd-c73176d4eae9","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:02:14.845816 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.845793    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:02:14.845941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.845828    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:02:14.878905 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.878871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:14.878905 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.878888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:14.879094 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:14.878875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:14.879094 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:14.878976    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:14.879094 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:14.879040    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:14.879207 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:14.879117    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:15.017287 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:15.017250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q" event={"ID":"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec","Type":"ContainerStarted","Data":"fc3b92aa478c6e3bafd921f247e033a57cf32a194e07a857b9fe4745fbddea26"}
Sep 16 12:02:15.018862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:15.018830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wh9cp" event={"ID":"a0480890-6eaf-4f5f-bd35-3d19ee846601","Type":"ContainerStarted","Data":"711be220810ae69af74d3d1ebd7e1df52756903ecddc24342a343ec81ad8dab7"}
Sep 16 12:02:15.035848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:15.035799    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wh9cp" podStartSLOduration=4.120948576 podStartE2EDuration="24.035782475s" totalImagesPullingTime="19.914833899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:15.035376399 +0000 UTC m=+24.672701249" watchObservedRunningTime="2026-09-16 12:02:15.035782475 +0000 UTC m=+24.673107340"
Sep 16 12:02:16.026245 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:16.025971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"005fb2fd255689077b31d67600cba8932657e249eabcd2434abff0de37667318"}
Sep 16 12:02:16.591593 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:16.591554    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:02:16.592370 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:16.592347    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:02:16.878719 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:16.878630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:16.878719 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:16.878630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:16.878941 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:16.878779    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:16.879083 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:16.879036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:16.879221 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:16.879197    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:16.879281 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:16.879211    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:17.031299 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:17.031263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q" event={"ID":"3355b66b-9d8b-4d6c-a316-fe38ff6b49ec","Type":"ContainerStarted","Data":"1f1a694a6139cf88b3cf574fb2e3226f3b35d4c2f226dd9208fbbdf0275b7e37"}
Sep 16 12:02:17.050366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:17.050311    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q584q" podStartSLOduration=3.485304234 podStartE2EDuration="26.050290385s" totalImagesPullingTime="22.564986151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:17.049992743 +0000 UTC m=+26.687317582" watchObservedRunningTime="2026-09-16 12:02:17.050290385 +0000 UTC m=+26.687615224"
Sep 16 12:02:17.436718 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:17.436683    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:02:17.437401 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:17.437379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hcxdk"
Sep 16 12:02:18.878255 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:18.878226    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:18.878255 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:18.878258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:18.878711 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:18.878226    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:18.878711 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:18.878355    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:18.878711 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:18.878382    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:18.878711 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:18.878505    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:19.039914 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:19.039880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" event={"ID":"2112a93a-a830-4c7b-a86e-657507d2d927","Type":"ContainerStarted","Data":"f866c464f86174b5982922ef0bbde1a3c4c18715ca30ad7c8c4aedb89a13aa9c"}
Sep 16 12:02:19.068939 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:19.068767    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm" podStartSLOduration=7.127724802 podStartE2EDuration="27.068751811s" totalImagesPullingTime="19.941027009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:19.068572775 +0000 UTC m=+28.705897614" watchObservedRunningTime="2026-09-16 12:02:19.068751811 +0000 UTC m=+28.706076648"
Sep 16 12:02:20.049430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.049390    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2238633-58be-4323-940e-5aa992160526" containerID="86a001d1d9e9643be20145d65cb061ad2969084c34ba9b4aa7be375920a44f2f" exitCode=0
Sep 16 12:02:20.050006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.049487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" event={"ID":"b2238633-58be-4323-940e-5aa992160526","Type":"ContainerDied","Data":"86a001d1d9e9643be20145d65cb061ad2969084c34ba9b4aa7be375920a44f2f"}
Sep 16 12:02:20.050006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.049989    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:02:20.050117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.050063    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:02:20.050117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.050108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:02:20.068366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.068336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:02:20.068726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.068708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:02:20.823228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.823195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j7tg4"]
Sep 16 12:02:20.823477 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.823336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:20.823477 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:20.823441    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:20.824849 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.824816    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8fst2"]
Sep 16 12:02:20.824961 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.824915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:20.825037 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:20.825012    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:20.832119 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.832060    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nhjbr"]
Sep 16 12:02:20.832223 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:20.832172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:20.832345 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:20.832313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:21.054027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:21.053749    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2238633-58be-4323-940e-5aa992160526" containerID="47d8bf0b56a9199e39860e5bebf32c57c1697fbbcaf3a621503fe1160f0fbe12" exitCode=0
Sep 16 12:02:21.054645 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:21.053829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" event={"ID":"b2238633-58be-4323-940e-5aa992160526","Type":"ContainerDied","Data":"47d8bf0b56a9199e39860e5bebf32c57c1697fbbcaf3a621503fe1160f0fbe12"}
Sep 16 12:02:21.878359 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:21.878335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:21.878485 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:21.878433    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:22.059822 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:22.059696    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2238633-58be-4323-940e-5aa992160526" containerID="cb41f2e8b86add5ae21e48f77cbeef0ec4d42a7ca0504c5321d093b87452d794" exitCode=0
Sep 16 12:02:22.059822 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:22.059766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" event={"ID":"b2238633-58be-4323-940e-5aa992160526","Type":"ContainerDied","Data":"cb41f2e8b86add5ae21e48f77cbeef0ec4d42a7ca0504c5321d093b87452d794"}
Sep 16 12:02:22.878830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:22.878797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:22.879018 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:22.878805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:22.879018 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:22.878898    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:22.879018 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:22.878981    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:23.878069 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:23.878031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:23.878447 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:23.878149    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:24.534178 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:24.534141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:24.534348 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.534303    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:24.534395 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.534373    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret podName:04c58447-61b8-400e-be4b-8519523f5788 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:56.534355526 +0000 UTC m=+66.171680341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret") pod "global-pull-secret-syncer-8fst2" (UID: "04c58447-61b8-400e-be4b-8519523f5788") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:24.634596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:24.634542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:24.634802 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.634745    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:24.634866 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.634805    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs podName:16ec4baa-3a59-4ba5-afd3-eafb08071c5a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:56.63479104 +0000 UTC m=+66.272115856 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs") pod "network-metrics-daemon-nhjbr" (UID: "16ec4baa-3a59-4ba5-afd3-eafb08071c5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:24.735665 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:24.735627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:24.735826 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.735769    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:02:24.735826 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.735783    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:02:24.735826 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.735793    2578 projected.go:196] Error preparing data for projected volume kube-api-access-v7ntk for pod openshift-network-diagnostics/network-check-target-j7tg4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:24.735923 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.735841    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk podName:7a847c41-95f4-40fb-aaed-ee5f5057584d nodeName:}" failed. No retries permitted until 2026-09-16 12:02:56.735828437 +0000 UTC m=+66.373153253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7ntk" (UniqueName: "kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk") pod "network-check-target-j7tg4" (UID: "7a847c41-95f4-40fb-aaed-ee5f5057584d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:24.878738 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:24.878659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:24.879080 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.878769    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7tg4" podUID="7a847c41-95f4-40fb-aaed-ee5f5057584d"
Sep 16 12:02:24.879080 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:24.878832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:24.879080 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:24.878952    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhjbr" podUID="16ec4baa-3a59-4ba5-afd3-eafb08071c5a"
Sep 16 12:02:25.878755 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:25.878729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:25.879198 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:25.878833    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fst2" podUID="04c58447-61b8-400e-be4b-8519523f5788"
Sep 16 12:02:26.169027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.169000    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-170.ec2.internal" event="NodeReady"
Sep 16 12:02:26.169182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.169124    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:02:26.206863 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.206835    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-v7sp6"]
Sep 16 12:02:26.229408 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.228533    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f9d96bb5-dj8r2"]
Sep 16 12:02:26.229408 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.228630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.231652 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.231626    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-th7jg\""
Sep 16 12:02:26.231813 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.231795    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 12:02:26.232062 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.232043    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 12:02:26.232627 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.232604    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.233432 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.233409    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.242346 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.242300    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 12:02:26.248753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.248730    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"]
Sep 16 12:02:26.249741 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.249722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.252330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.252308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:02:26.252880 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.252859    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:02:26.252880 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.252873    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:02:26.253331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.253198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mpvqx\""
Sep 16 12:02:26.259557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.259138    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:02:26.271313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.270550    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-v7sp6"]
Sep 16 12:02:26.271313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.270584    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"]
Sep 16 12:02:26.271313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.270603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f9d96bb5-dj8r2"]
Sep 16 12:02:26.271313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.270620    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"]
Sep 16 12:02:26.271313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.270619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.273609 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.273590    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.273707 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.273643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gng5k\""
Sep 16 12:02:26.274793 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.274769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 12:02:26.274793 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.274784    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.274934 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.274806    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 12:02:26.292721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.292696    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"]
Sep 16 12:02:26.292823 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.292813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.295893 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.295871    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 12:02:26.296438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.296291    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.296438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.296303    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.296438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.296311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 12:02:26.296438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.296320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-v8h75\""
Sep 16 12:02:26.312733 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.312711    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qz5zw"]
Sep 16 12:02:26.312861 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.312845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:26.315354 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.315329    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 12:02:26.315354 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.315342    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.315682 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.315665    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.315880 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.315865    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4c4mq\""
Sep 16 12:02:26.326726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.326709    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-544f785cb6-6wc5m"]
Sep 16 12:02:26.326850 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.326836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.329499 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.329481    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:02:26.329499 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.329488    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8hlmq\""
Sep 16 12:02:26.329658 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.329488    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:02:26.346909 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.346871    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"]
Sep 16 12:02:26.348351 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.348331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.350414 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqzv\" (UniqueName: \"kubernetes.io/projected/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-kube-api-access-lsqzv\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.350540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-installation-pull-secrets\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.350540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2tr\" (UniqueName: \"kubernetes.io/projected/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-kube-api-access-mp2tr\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.350540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1664da60-8154-46fe-9ce7-fd3656a0e422-ca-trust-extracted\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.350540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.350745 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.350745 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-certificates\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.350835 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-bound-sa-token\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.350835 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-service-ca-bundle\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.350835 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.350975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-image-registry-private-configuration\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.350975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-trusted-ca\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.350975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-tmp\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.350975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfm6f\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-kube-api-access-hfm6f\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.350975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.350975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-serving-cert\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.351246 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.350995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-snapshots\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.351999 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.351979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 12:02:26.352112 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.352072    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 12:02:26.352198 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.352181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 12:02:26.352272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.352250    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lxplz\""
Sep 16 12:02:26.352401 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.352384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 12:02:26.352525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.352419    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.352656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.352637    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.369564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.369540    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg"]
Sep 16 12:02:26.369686 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.369546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.372383 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.372361    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.372495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.372373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 12:02:26.372495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.372479    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-npwvp\""
Sep 16 12:02:26.372619 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.372418    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.372619 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.372382    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 12:02:26.389584 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.389566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg"
Sep 16 12:02:26.389828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.389570    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8ls9j"]
Sep 16 12:02:26.392329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.392310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.392613 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.392596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.392797 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.392782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6dxgh\""
Sep 16 12:02:26.409703 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.409682    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qv7vb"]
Sep 16 12:02:26.409831 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.409815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.412200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.412173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 12:02:26.412416 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.412398    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.412607 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.412588    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wpflc\""
Sep 16 12:02:26.412794 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.412775    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 12:02:26.412868 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.412591    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.417614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.417594    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 12:02:26.423888 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.423870    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-82xgp"]
Sep 16 12:02:26.424086 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.424069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qv7vb"
Sep 16 12:02:26.426830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.426799    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:02:26.426927 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.426873    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.426994 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.426961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rcl6b\""
Sep 16 12:02:26.427047 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.427012    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.446033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"]
Sep 16 12:02:26.446033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"]
Sep 16 12:02:26.446175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qv7vb"]
Sep 16 12:02:26.446175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg"]
Sep 16 12:02:26.446175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446103    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"]
Sep 16 12:02:26.446175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446121    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8ls9j"]
Sep 16 12:02:26.446175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446138    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qz5zw"]
Sep 16 12:02:26.446175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-82xgp"
Sep 16 12:02:26.446175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-82xgp"]
Sep 16 12:02:26.446463 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.446184    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-544f785cb6-6wc5m"]
Sep 16 12:02:26.449217 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.449002    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vp9wp\""
Sep 16 12:02:26.449217 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.449070    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:02:26.449217 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.449005    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:02:26.451445 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-installation-pull-secrets\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.451553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.451553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.451553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-certificates\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.451704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-config\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.451704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acd2a478-2880-4d82-8a20-70e1e6a538c0-config-volume\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.451704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.451704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b3e46af-da21-45fc-8b38-f6754a2a7c70-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.451704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-serving-cert\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.451853 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-image-registry-private-configuration\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.451853 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-tmp\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.451853 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfm6f\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-kube-api-access-hfm6f\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.451853 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2tr\" (UniqueName: \"kubernetes.io/projected/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-kube-api-access-mp2tr\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.451853 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-default-certificate\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.452065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.452065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.452065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b3e46af-da21-45fc-8b38-f6754a2a7c70-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.452065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.452065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd2a478-2880-4d82-8a20-70e1e6a538c0-tmp-dir\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.452065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.451989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1664da60-8154-46fe-9ce7-fd3656a0e422-ca-trust-extracted\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.452065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rr79\" (UniqueName: \"kubernetes.io/projected/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-kube-api-access-2rr79\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.452065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqzv\" (UniqueName: \"kubernetes.io/projected/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-kube-api-access-lsqzv\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-stats-auth\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-bound-sa-token\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-serving-cert\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-service-ca-bundle\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7285h\" (UniqueName: \"kubernetes.io/projected/de22da40-9f25-4233-87ff-9fbc536d1a7e-kube-api-access-7285h\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckssk\" (UniqueName: \"kubernetes.io/projected/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-kube-api-access-ckssk\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-trusted-ca\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxnj\" (UniqueName: \"kubernetes.io/projected/1b3e46af-da21-45fc-8b38-f6754a2a7c70-kube-api-access-kdxnj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp2x\" (UniqueName: \"kubernetes.io/projected/acd2a478-2880-4d82-8a20-70e1e6a538c0-kube-api-access-gvp2x\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.452484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.452381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-snapshots\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.453076 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.453058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-snapshots\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.453290    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.453308    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9d96bb5-dj8r2: secret "image-registry-tls" not found
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.453367    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls podName:1664da60-8154-46fe-9ce7-fd3656a0e422 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:26.953347596 +0000 UTC m=+36.590672426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls") pod "image-registry-6f9d96bb5-dj8r2" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422") : secret "image-registry-tls" not found
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.453433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1664da60-8154-46fe-9ce7-fd3656a0e422-ca-trust-extracted\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.453464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-tmp\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.453978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.454182    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.454246    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls podName:5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:26.954225666 +0000 UTC m=+36.591550491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rx5ch" (UID: "5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:26.454610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.454349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.454970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.454816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-certificates\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.455303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.455280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-service-ca-bundle\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.455398 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.455375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-trusted-ca\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.458451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.458411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-serving-cert\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.458451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.458426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-installation-pull-secrets\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.459116 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.459092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-image-registry-private-configuration\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.466739 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.466715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2tr\" (UniqueName: \"kubernetes.io/projected/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-kube-api-access-mp2tr\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.466840 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.466816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqzv\" (UniqueName: \"kubernetes.io/projected/14f56708-c1b2-4ac8-8a6b-3eba48cfb62c-kube-api-access-lsqzv\") pod \"insights-operator-5844b856db-v7sp6\" (UID: \"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c\") " pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.466896 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.466870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfm6f\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-kube-api-access-hfm6f\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.467861 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.467843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-bound-sa-token\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.544420 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.544381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-v7sp6"
Sep 16 12:02:26.553228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp2x\" (UniqueName: \"kubernetes.io/projected/acd2a478-2880-4d82-8a20-70e1e6a538c0-kube-api-access-gvp2x\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.553365 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-config\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.553365 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acd2a478-2880-4d82-8a20-70e1e6a538c0-config-volume\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.553473 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b3e46af-da21-45fc-8b38-f6754a2a7c70-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.553473 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrq4\" (UniqueName: \"kubernetes.io/projected/3ef15515-9c47-49b7-bd0e-9832d387fee7-kube-api-access-fkrq4\") pod \"volume-data-source-validator-86695b7cc7-pkctg\" (UID: \"3ef15515-9c47-49b7-bd0e-9832d387fee7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg"
Sep 16 12:02:26.553473 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-serving-cert\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.553473 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-serving-cert\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.553473 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wz8\" (UniqueName: \"kubernetes.io/projected/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-kube-api-access-n9wz8\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.553737 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v627s\" (UniqueName: \"kubernetes.io/projected/9b44461a-fdcb-4a38-83d3-e72418354565-kube-api-access-v627s\") pod \"ingress-canary-qv7vb\" (UID: \"9b44461a-fdcb-4a38-83d3-e72418354565\") " pod="openshift-ingress-canary/ingress-canary-qv7vb"
Sep 16 12:02:26.553737 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-default-certificate\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.553737 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.553737 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.553737 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b3e46af-da21-45fc-8b38-f6754a2a7c70-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.553737 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.553675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.554049 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.553836    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:26.554049 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.553859    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:27.053839806 +0000 UTC m=+36.691164625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : configmap references non-existent config key: service-ca.crt
Sep 16 12:02:26.554049 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.553905    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls podName:acd2a478-2880-4d82-8a20-70e1e6a538c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:27.05388943 +0000 UTC m=+36.691214247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls") pod "dns-default-qz5zw" (UID: "acd2a478-2880-4d82-8a20-70e1e6a538c0") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:26.554224 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-config\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.554224 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd2a478-2880-4d82-8a20-70e1e6a538c0-tmp-dir\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.554224 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.554148    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:02:26.554224 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acd2a478-2880-4d82-8a20-70e1e6a538c0-config-volume\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.554224 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.554216    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:27.054202014 +0000 UTC m=+36.691526833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : secret "router-metrics-certs-default" not found
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rr79\" (UniqueName: \"kubernetes.io/projected/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-kube-api-access-2rr79\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b3e46af-da21-45fc-8b38-f6754a2a7c70-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-stats-auth\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-config\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7285h\" (UniqueName: \"kubernetes.io/projected/de22da40-9f25-4233-87ff-9fbc536d1a7e-kube-api-access-7285h\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckssk\" (UniqueName: \"kubernetes.io/projected/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-kube-api-access-ckssk\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxnj\" (UniqueName: \"kubernetes.io/projected/1b3e46af-da21-45fc-8b38-f6754a2a7c70-kube-api-access-kdxnj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.554461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-trusted-ca\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.554900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vxsd\" (UniqueName: \"kubernetes.io/projected/b64b3bac-9362-4dc7-b81a-c9cec6118ca1-kube-api-access-7vxsd\") pod \"network-check-source-6f8b768455-82xgp\" (UID: \"b64b3bac-9362-4dc7-b81a-c9cec6118ca1\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-82xgp"
Sep 16 12:02:26.554900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b44461a-fdcb-4a38-83d3-e72418354565-cert\") pod \"ingress-canary-qv7vb\" (UID: \"9b44461a-fdcb-4a38-83d3-e72418354565\") " pod="openshift-ingress-canary/ingress-canary-qv7vb"
Sep 16 12:02:26.554900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.554597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd2a478-2880-4d82-8a20-70e1e6a538c0-tmp-dir\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.554900 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.554733    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:02:26.554900 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.554795    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls podName:228f1c38-5d40-47ec-9cf4-b11c93ce79aa nodeName:}" failed. No retries permitted until 2026-09-16 12:02:27.054779055 +0000 UTC m=+36.692103887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kmk6m" (UID: "228f1c38-5d40-47ec-9cf4-b11c93ce79aa") : secret "samples-operator-tls" not found
Sep 16 12:02:26.556366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.556338    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b3e46af-da21-45fc-8b38-f6754a2a7c70-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.556470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.556349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-serving-cert\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.556782 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.556763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-default-certificate\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.558465 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.558443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-stats-auth\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.564693 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.564633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckssk\" (UniqueName: \"kubernetes.io/projected/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-kube-api-access-ckssk\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:26.565446 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.565423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rr79\" (UniqueName: \"kubernetes.io/projected/fdeaddaa-5009-4d0b-99b6-57f6bb8b219a-kube-api-access-2rr79\") pod \"service-ca-operator-cf7c776b7-jsvk6\" (UID: \"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.565764 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.565719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxnj\" (UniqueName: \"kubernetes.io/projected/1b3e46af-da21-45fc-8b38-f6754a2a7c70-kube-api-access-kdxnj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x8hnx\" (UID: \"1b3e46af-da21-45fc-8b38-f6754a2a7c70\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.566061 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.566043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp2x\" (UniqueName: \"kubernetes.io/projected/acd2a478-2880-4d82-8a20-70e1e6a538c0-kube-api-access-gvp2x\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:26.566125 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.566064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7285h\" (UniqueName: \"kubernetes.io/projected/de22da40-9f25-4233-87ff-9fbc536d1a7e-kube-api-access-7285h\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:26.602688 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.602621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"
Sep 16 12:02:26.655874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.655839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrq4\" (UniqueName: \"kubernetes.io/projected/3ef15515-9c47-49b7-bd0e-9832d387fee7-kube-api-access-fkrq4\") pod \"volume-data-source-validator-86695b7cc7-pkctg\" (UID: \"3ef15515-9c47-49b7-bd0e-9832d387fee7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg"
Sep 16 12:02:26.656048 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.655884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-serving-cert\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.656048 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.655913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wz8\" (UniqueName: \"kubernetes.io/projected/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-kube-api-access-n9wz8\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.656048 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.655935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v627s\" (UniqueName: \"kubernetes.io/projected/9b44461a-fdcb-4a38-83d3-e72418354565-kube-api-access-v627s\") pod \"ingress-canary-qv7vb\" (UID: \"9b44461a-fdcb-4a38-83d3-e72418354565\") " pod="openshift-ingress-canary/ingress-canary-qv7vb"
Sep 16 12:02:26.656197 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.656155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-config\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.656271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.656196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-trusted-ca\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.656271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.656215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vxsd\" (UniqueName: \"kubernetes.io/projected/b64b3bac-9362-4dc7-b81a-c9cec6118ca1-kube-api-access-7vxsd\") pod \"network-check-source-6f8b768455-82xgp\" (UID: \"b64b3bac-9362-4dc7-b81a-c9cec6118ca1\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-82xgp"
Sep 16 12:02:26.656271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.656237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b44461a-fdcb-4a38-83d3-e72418354565-cert\") pod \"ingress-canary-qv7vb\" (UID: \"9b44461a-fdcb-4a38-83d3-e72418354565\") " pod="openshift-ingress-canary/ingress-canary-qv7vb"
Sep 16 12:02:26.657093 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.657050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-config\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.657197 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.657117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-trusted-ca\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.659104 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.659082    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-serving-cert\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.659392 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.659376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b44461a-fdcb-4a38-83d3-e72418354565-cert\") pod \"ingress-canary-qv7vb\" (UID: \"9b44461a-fdcb-4a38-83d3-e72418354565\") " pod="openshift-ingress-canary/ingress-canary-qv7vb"
Sep 16 12:02:26.665222 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.665201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vxsd\" (UniqueName: \"kubernetes.io/projected/b64b3bac-9362-4dc7-b81a-c9cec6118ca1-kube-api-access-7vxsd\") pod \"network-check-source-6f8b768455-82xgp\" (UID: \"b64b3bac-9362-4dc7-b81a-c9cec6118ca1\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-82xgp"
Sep 16 12:02:26.665318 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.665245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wz8\" (UniqueName: \"kubernetes.io/projected/d2bd3fc1-6334-453c-8ca1-048c8b77bc17-kube-api-access-n9wz8\") pod \"console-operator-696884fb9c-8ls9j\" (UID: \"d2bd3fc1-6334-453c-8ca1-048c8b77bc17\") " pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.665460 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.665439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v627s\" (UniqueName: \"kubernetes.io/projected/9b44461a-fdcb-4a38-83d3-e72418354565-kube-api-access-v627s\") pod \"ingress-canary-qv7vb\" (UID: \"9b44461a-fdcb-4a38-83d3-e72418354565\") " pod="openshift-ingress-canary/ingress-canary-qv7vb"
Sep 16 12:02:26.665806 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.665764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrq4\" (UniqueName: \"kubernetes.io/projected/3ef15515-9c47-49b7-bd0e-9832d387fee7-kube-api-access-fkrq4\") pod \"volume-data-source-validator-86695b7cc7-pkctg\" (UID: \"3ef15515-9c47-49b7-bd0e-9832d387fee7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg"
Sep 16 12:02:26.679233 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.679210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"
Sep 16 12:02:26.699606 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.699582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg"
Sep 16 12:02:26.719970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.719950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:26.733790 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.733767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qv7vb"
Sep 16 12:02:26.756044 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.755910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-82xgp"
Sep 16 12:02:26.878871 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.878788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:26.879320 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.878792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:26.881997 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.881972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:02:26.882154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.882059    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qzcvp\""
Sep 16 12:02:26.882333 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.882311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pcfgd\""
Sep 16 12:02:26.959848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.959812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:26.959848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:26.959851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:26.960067 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.959993    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:26.960067 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.960008    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:26.960067 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.960023    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9d96bb5-dj8r2: secret "image-registry-tls" not found
Sep 16 12:02:26.960067 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.960067    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls podName:5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:27.960045496 +0000 UTC m=+37.597370315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rx5ch" (UID: "5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:26.960252 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:26.960131    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls podName:1664da60-8154-46fe-9ce7-fd3656a0e422 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:27.96012094 +0000 UTC m=+37.597445757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls") pod "image-registry-6f9d96bb5-dj8r2" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422") : secret "image-registry-tls" not found
Sep 16 12:02:27.061397 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:27.061357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:27.061397 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:27.061405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:27.061430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:27.061473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.061545    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.061545    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.061631    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:28.061609863 +0000 UTC m=+37.698934703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : secret "router-metrics-certs-default" not found
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.061643    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.061650    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:28.061640168 +0000 UTC m=+37.698964991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : configmap references non-existent config key: service-ca.crt
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.061712    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls podName:acd2a478-2880-4d82-8a20-70e1e6a538c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:28.06169492 +0000 UTC m=+37.699019743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls") pod "dns-default-qz5zw" (UID: "acd2a478-2880-4d82-8a20-70e1e6a538c0") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:27.061722 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.061731    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls podName:228f1c38-5d40-47ec-9cf4-b11c93ce79aa nodeName:}" failed. No retries permitted until 2026-09-16 12:02:28.061718513 +0000 UTC m=+37.699043334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kmk6m" (UID: "228f1c38-5d40-47ec-9cf4-b11c93ce79aa") : secret "samples-operator-tls" not found
Sep 16 12:02:27.878496 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:27.878037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:27.882860 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:27.882698    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:02:27.970204 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:27.970022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:27.970322 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:27.970227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:27.970322 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.970268    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:27.970322 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.970318    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:27.970481 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.970329    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9d96bb5-dj8r2: secret "image-registry-tls" not found
Sep 16 12:02:27.970481 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.970333    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls podName:5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:29.970314266 +0000 UTC m=+39.607639098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rx5ch" (UID: "5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:27.970481 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:27.970360    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls podName:1664da60-8154-46fe-9ce7-fd3656a0e422 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:29.970350228 +0000 UTC m=+39.607675044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls") pod "image-registry-6f9d96bb5-dj8r2" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422") : secret "image-registry-tls" not found
Sep 16 12:02:28.070927 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.070843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:28.070927 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.070882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:28.070927 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.070901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:28.071176 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.070935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:28.071176 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:28.071000    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:02:28.071176 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:28.071013    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:28.071176 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:28.071036    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:02:28.071176 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:28.071054    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:30.071031163 +0000 UTC m=+39.708355994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : configmap references non-existent config key: service-ca.crt
Sep 16 12:02:28.071176 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:28.071075    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:30.071069161 +0000 UTC m=+39.708393976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : secret "router-metrics-certs-default" not found
Sep 16 12:02:28.071176 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:28.071086    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls podName:acd2a478-2880-4d82-8a20-70e1e6a538c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:30.071080668 +0000 UTC m=+39.708405484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls") pod "dns-default-qz5zw" (UID: "acd2a478-2880-4d82-8a20-70e1e6a538c0") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:28.071176 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:28.071095    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls podName:228f1c38-5d40-47ec-9cf4-b11c93ce79aa nodeName:}" failed. No retries permitted until 2026-09-16 12:02:30.071090533 +0000 UTC m=+39.708415349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kmk6m" (UID: "228f1c38-5d40-47ec-9cf4-b11c93ce79aa") : secret "samples-operator-tls" not found
Sep 16 12:02:28.152750 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.152721    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8ls9j"]
Sep 16 12:02:28.156693 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.156604    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qv7vb"]
Sep 16 12:02:28.157152 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:28.157085    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bd3fc1_6334_453c_8ca1_048c8b77bc17.slice/crio-2afbaec4213427af02b10168900bf8fe6400b466d7cb2f00cfd9805fe286d36f WatchSource:0}: Error finding container 2afbaec4213427af02b10168900bf8fe6400b466d7cb2f00cfd9805fe286d36f: Status 404 returned error can't find the container with id 2afbaec4213427af02b10168900bf8fe6400b466d7cb2f00cfd9805fe286d36f
Sep 16 12:02:28.162121 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.162014    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx"]
Sep 16 12:02:28.162185 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:28.162152    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b44461a_fdcb_4a38_83d3_e72418354565.slice/crio-cc4aa0b7339de941cc0ded952ad5ae75359e42d36ab04286609b798c42f27c46 WatchSource:0}: Error finding container cc4aa0b7339de941cc0ded952ad5ae75359e42d36ab04286609b798c42f27c46: Status 404 returned error can't find the container with id cc4aa0b7339de941cc0ded952ad5ae75359e42d36ab04286609b798c42f27c46
Sep 16 12:02:28.166830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.164001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-v7sp6"]
Sep 16 12:02:28.166830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.165128    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6"]
Sep 16 12:02:28.166830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.166794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg"]
Sep 16 12:02:28.167891 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:28.167866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-82xgp"]
Sep 16 12:02:28.168056 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:28.168036    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f56708_c1b2_4ac8_8a6b_3eba48cfb62c.slice/crio-6d86756b1b3364a5aada8ccb6b7110813737bcd0c90e955d5ea1510bf97088a8 WatchSource:0}: Error finding container 6d86756b1b3364a5aada8ccb6b7110813737bcd0c90e955d5ea1510bf97088a8: Status 404 returned error can't find the container with id 6d86756b1b3364a5aada8ccb6b7110813737bcd0c90e955d5ea1510bf97088a8
Sep 16 12:02:28.170894 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:28.170481    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64b3bac_9362_4dc7_b81a_c9cec6118ca1.slice/crio-3c07184a53917fc35ae535b9a1871125ec82e14a8905bc3f411446644ce396ec WatchSource:0}: Error finding container 3c07184a53917fc35ae535b9a1871125ec82e14a8905bc3f411446644ce396ec: Status 404 returned error can't find the container with id 3c07184a53917fc35ae535b9a1871125ec82e14a8905bc3f411446644ce396ec
Sep 16 12:02:28.171727 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:28.171616    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdeaddaa_5009_4d0b_99b6_57f6bb8b219a.slice/crio-efab44591833930f510cd752921f8181252b0b66ddb51b395fb5cc0d45567a7f WatchSource:0}: Error finding container efab44591833930f510cd752921f8181252b0b66ddb51b395fb5cc0d45567a7f: Status 404 returned error can't find the container with id efab44591833930f510cd752921f8181252b0b66ddb51b395fb5cc0d45567a7f
Sep 16 12:02:29.084311 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.084271    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2238633-58be-4323-940e-5aa992160526" containerID="4283781a0c5879ed368f9cbae3b6423ab67ff99858670e0a5ffe6b4bc62aba9d" exitCode=0
Sep 16 12:02:29.085694 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.084367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" event={"ID":"b2238633-58be-4323-940e-5aa992160526","Type":"ContainerDied","Data":"4283781a0c5879ed368f9cbae3b6423ab67ff99858670e0a5ffe6b4bc62aba9d"}
Sep 16 12:02:29.089899 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.089868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg" event={"ID":"3ef15515-9c47-49b7-bd0e-9832d387fee7","Type":"ContainerStarted","Data":"56f7bb1f48d58e624ede6a4858402c8d3ac7e82b6ecc83a745294a8993f84c4f"}
Sep 16 12:02:29.092557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.092507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-82xgp" event={"ID":"b64b3bac-9362-4dc7-b81a-c9cec6118ca1","Type":"ContainerStarted","Data":"3c07184a53917fc35ae535b9a1871125ec82e14a8905bc3f411446644ce396ec"}
Sep 16 12:02:29.097818 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.097784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qv7vb" event={"ID":"9b44461a-fdcb-4a38-83d3-e72418354565","Type":"ContainerStarted","Data":"cc4aa0b7339de941cc0ded952ad5ae75359e42d36ab04286609b798c42f27c46"}
Sep 16 12:02:29.101421 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.101396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" event={"ID":"d2bd3fc1-6334-453c-8ca1-048c8b77bc17","Type":"ContainerStarted","Data":"2afbaec4213427af02b10168900bf8fe6400b466d7cb2f00cfd9805fe286d36f"}
Sep 16 12:02:29.105593 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.105545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-v7sp6" event={"ID":"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c","Type":"ContainerStarted","Data":"6d86756b1b3364a5aada8ccb6b7110813737bcd0c90e955d5ea1510bf97088a8"}
Sep 16 12:02:29.107248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.107149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx" event={"ID":"1b3e46af-da21-45fc-8b38-f6754a2a7c70","Type":"ContainerStarted","Data":"7cbab6bf641d257b88f5c7726f3e37ca79001809658d87e1bd1a9b19da3345df"}
Sep 16 12:02:29.109273 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.109217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6" event={"ID":"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a","Type":"ContainerStarted","Data":"efab44591833930f510cd752921f8181252b0b66ddb51b395fb5cc0d45567a7f"}
Sep 16 12:02:29.993467 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.993421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:29.993646 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:29.993471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:29.993875 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:29.993824    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:29.993875 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:29.993845    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9d96bb5-dj8r2: secret "image-registry-tls" not found
Sep 16 12:02:29.994008 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:29.993904    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls podName:1664da60-8154-46fe-9ce7-fd3656a0e422 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:33.993884307 +0000 UTC m=+43.631209129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls") pod "image-registry-6f9d96bb5-dj8r2" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422") : secret "image-registry-tls" not found
Sep 16 12:02:29.994383 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:29.994359    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:29.994470 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:29.994421    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls podName:5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:33.994405031 +0000 UTC m=+43.631729869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rx5ch" (UID: "5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:30.095277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:30.095210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:30.095753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:30.095312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:30.095753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:30.095345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:30.095753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:30.095385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:30.095753 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:30.095603    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:02:30.095753 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:30.095668    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls podName:228f1c38-5d40-47ec-9cf4-b11c93ce79aa nodeName:}" failed. No retries permitted until 2026-09-16 12:02:34.095647191 +0000 UTC m=+43.732972012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kmk6m" (UID: "228f1c38-5d40-47ec-9cf4-b11c93ce79aa") : secret "samples-operator-tls" not found
Sep 16 12:02:30.096537 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:30.096071    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:02:30.096537 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:30.096123    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:34.096108155 +0000 UTC m=+43.733432975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : secret "router-metrics-certs-default" not found
Sep 16 12:02:30.096537 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:30.096182    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:30.096537 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:30.096212    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls podName:acd2a478-2880-4d82-8a20-70e1e6a538c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:34.096201488 +0000 UTC m=+43.733526307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls") pod "dns-default-qz5zw" (UID: "acd2a478-2880-4d82-8a20-70e1e6a538c0") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:30.096537 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:30.096275    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:34.096263985 +0000 UTC m=+43.733588816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : configmap references non-existent config key: service-ca.crt
Sep 16 12:02:30.122445 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:30.122412    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2238633-58be-4323-940e-5aa992160526" containerID="c0b609b1a870161d9beb034d58e09560f4e9cd2eab693a541f91eb1b66ca4acb" exitCode=0
Sep 16 12:02:30.122607 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:30.122500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" event={"ID":"b2238633-58be-4323-940e-5aa992160526","Type":"ContainerDied","Data":"c0b609b1a870161d9beb034d58e09560f4e9cd2eab693a541f91eb1b66ca4acb"}
Sep 16 12:02:34.034398 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:34.034169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:34.034859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:34.034418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:34.034859 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.034331    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:34.034859 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.034560    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls podName:5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:42.034539636 +0000 UTC m=+51.671864461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rx5ch" (UID: "5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:34.034859 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.034640    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:34.034859 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.034661    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9d96bb5-dj8r2: secret "image-registry-tls" not found
Sep 16 12:02:34.034859 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.034724    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls podName:1664da60-8154-46fe-9ce7-fd3656a0e422 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:42.034705999 +0000 UTC m=+51.672030839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls") pod "image-registry-6f9d96bb5-dj8r2" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422") : secret "image-registry-tls" not found
Sep 16 12:02:34.135638 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:34.135603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:34.135824 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:34.135644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:34.135824 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:34.135668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:34.135824 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:34.135706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:34.135824 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.135769    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:34.135824 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.135772    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:02:34.136043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.135840    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:42.135820353 +0000 UTC m=+51.773145172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : secret "router-metrics-certs-default" not found
Sep 16 12:02:34.136043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.135858    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls podName:acd2a478-2880-4d82-8a20-70e1e6a538c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:42.135849025 +0000 UTC m=+51.773173841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls") pod "dns-default-qz5zw" (UID: "acd2a478-2880-4d82-8a20-70e1e6a538c0") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:34.136043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.135860    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:02:34.136043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.135873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:42.135864808 +0000 UTC m=+51.773189625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : configmap references non-existent config key: service-ca.crt
Sep 16 12:02:34.136043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:34.135906    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls podName:228f1c38-5d40-47ec-9cf4-b11c93ce79aa nodeName:}" failed. No retries permitted until 2026-09-16 12:02:42.135891307 +0000 UTC m=+51.773216127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kmk6m" (UID: "228f1c38-5d40-47ec-9cf4-b11c93ce79aa") : secret "samples-operator-tls" not found
Sep 16 12:02:36.146816 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:36.146698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" event={"ID":"b2238633-58be-4323-940e-5aa992160526","Type":"ContainerStarted","Data":"3c5b3cd3809eb057c23c4fd82e270923be2593d48c92890d8ccc66e78ca956b2"}
Sep 16 12:02:36.149456 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:36.149432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg" event={"ID":"3ef15515-9c47-49b7-bd0e-9832d387fee7","Type":"ContainerStarted","Data":"aabe20ff74cd01f378b3c1d78c58501c26cf89a96a48ada74884c2196eb97bea"}
Sep 16 12:02:36.191539 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:36.189308    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2xqd9" podStartSLOduration=12.613474873 podStartE2EDuration="45.189285574s" totalImagesPullingTime="32.575810701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:36.170703946 +0000 UTC m=+45.808028785" watchObservedRunningTime="2026-09-16 12:02:36.189285574 +0000 UTC m=+45.826610413"
Sep 16 12:02:36.191539 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:36.190154    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pkctg" podStartSLOduration=29.458875567 podStartE2EDuration="33.190137826s" totalImagesPullingTime="3.731262259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:36.188679312 +0000 UTC m=+45.826004152" watchObservedRunningTime="2026-09-16 12:02:36.190137826 +0000 UTC m=+45.827462664"
Sep 16 12:02:37.156689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.156652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-v7sp6" event={"ID":"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c","Type":"ContainerStarted","Data":"3e9219d504afeafa746891ff0053403639d135d91a0776ceb79ebd8293febf38"}
Sep 16 12:02:37.158449 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.158416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx" event={"ID":"1b3e46af-da21-45fc-8b38-f6754a2a7c70","Type":"ContainerStarted","Data":"0ad98db108ff43f4c737f1d7bbddf258df96bc72de3c13106f5bb1967de5da39"}
Sep 16 12:02:37.160025 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.159999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6" event={"ID":"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a","Type":"ContainerStarted","Data":"481532cfdf9f3d3f71270a0248a4841520f801ce9ed91e9d91982e3b02e8c729"}
Sep 16 12:02:37.161699 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.161651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-82xgp" event={"ID":"b64b3bac-9362-4dc7-b81a-c9cec6118ca1","Type":"ContainerStarted","Data":"41122c2c515996af8e869a09dc6e7b18ff9ea7ce54891d5b931ebb1a2fbee06d"}
Sep 16 12:02:37.163376 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.163353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qv7vb" event={"ID":"9b44461a-fdcb-4a38-83d3-e72418354565","Type":"ContainerStarted","Data":"672de0fb3366e181aa02155886297e215a38e83db548b595b5157ca3e21674e9"}
Sep 16 12:02:37.165176 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.165157    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/0.log"
Sep 16 12:02:37.165263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.165197    2578 generic.go:356] "Generic (PLEG): container finished" podID="d2bd3fc1-6334-453c-8ca1-048c8b77bc17" containerID="263f0f9e80d33475f91e5588f97c6b533fa9cb06ade5ef819c7e40f0a948ab36" exitCode=255
Sep 16 12:02:37.165327 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.165283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" event={"ID":"d2bd3fc1-6334-453c-8ca1-048c8b77bc17","Type":"ContainerDied","Data":"263f0f9e80d33475f91e5588f97c6b533fa9cb06ade5ef819c7e40f0a948ab36"}
Sep 16 12:02:37.165473 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.165458    2578 scope.go:117] "RemoveContainer" containerID="263f0f9e80d33475f91e5588f97c6b533fa9cb06ade5ef819c7e40f0a948ab36"
Sep 16 12:02:37.173914 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.173874    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-v7sp6" podStartSLOduration=27.325114591 podStartE2EDuration="35.173858248s" totalImagesPullingTime="7.848743657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:37.173464605 +0000 UTC m=+46.810789436" watchObservedRunningTime="2026-09-16 12:02:37.173858248 +0000 UTC m=+46.811183086"
Sep 16 12:02:37.190618 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.190575    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx" podStartSLOduration=26.340806255 podStartE2EDuration="34.190557523s" totalImagesPullingTime="7.849751268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:37.189176803 +0000 UTC m=+46.826501642" watchObservedRunningTime="2026-09-16 12:02:37.190557523 +0000 UTC m=+46.827882362"
Sep 16 12:02:37.209601 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.209554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qv7vb" podStartSLOduration=3.352857098 podStartE2EDuration="11.209536058s" totalImagesPullingTime="7.85667896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:37.208263524 +0000 UTC m=+46.845588364" watchObservedRunningTime="2026-09-16 12:02:37.209536058 +0000 UTC m=+46.846860894"
Sep 16 12:02:37.228709 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.228665    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-82xgp" podStartSLOduration=26.36883388 podStartE2EDuration="34.228646158s" totalImagesPullingTime="7.859812278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:37.227334418 +0000 UTC m=+46.864659260" watchObservedRunningTime="2026-09-16 12:02:37.228646158 +0000 UTC m=+46.865970997"
Sep 16 12:02:37.249405 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.248104    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6" podStartSLOduration=26.400327222 podStartE2EDuration="34.248077781s" totalImagesPullingTime="7.847750559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:37.247445484 +0000 UTC m=+46.884770335" watchObservedRunningTime="2026-09-16 12:02:37.248077781 +0000 UTC m=+46.885402620"
Sep 16 12:02:37.488688 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.488652    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l8q8l"]
Sep 16 12:02:37.501632 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.501549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.502589 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.502495    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8q8l"]
Sep 16 12:02:37.504307 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.504286    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4nl5\""
Sep 16 12:02:37.504573 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.504555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:02:37.504866 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.504849    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:02:37.565592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.565571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-data-volume\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.565704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.565604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.565753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.565722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxvp\" (UniqueName: \"kubernetes.io/projected/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-api-access-9pxvp\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.565803 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.565785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-crio-socket\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.565854 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.565838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.666244 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.666205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.666379 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.666324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-data-volume\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.666379 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:37.666348    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:37.666379 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.666361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.666537 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:37.666409    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:38.166387448 +0000 UTC m=+47.803712278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:37.666537 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:37.666451    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:02:37.666537 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.666457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxvp\" (UniqueName: \"kubernetes.io/projected/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-api-access-9pxvp\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.666537 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:37.666502    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:38.166486487 +0000 UTC m=+47.803811310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : configmap "kube-rbac-proxy" not found
Sep 16 12:02:37.666776 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.666568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-crio-socket\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.666776 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.666718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-data-volume\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.666869 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.666773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-crio-socket\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:37.679137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:37.679112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxvp\" (UniqueName: \"kubernetes.io/projected/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-api-access-9pxvp\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:38.169615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:38.169573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:38.170043 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:38.169684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:38.170043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:38.169722    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:38.170043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:38.169732    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:02:38.170043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:38.169795    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:39.169775214 +0000 UTC m=+48.807100044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : configmap "kube-rbac-proxy" not found
Sep 16 12:02:38.170043 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:38.169811    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:39.169802849 +0000 UTC m=+48.807127665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:38.170443 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:38.170426    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:02:38.170949 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:38.170931    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/0.log"
Sep 16 12:02:38.171043 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:38.170978    2578 generic.go:356] "Generic (PLEG): container finished" podID="d2bd3fc1-6334-453c-8ca1-048c8b77bc17" containerID="9ea4b632a5ff71cc4f0808f36e7e2e27cc30d597e6548a0b52edaa45718500f7" exitCode=255
Sep 16 12:02:38.171120 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:38.171094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" event={"ID":"d2bd3fc1-6334-453c-8ca1-048c8b77bc17","Type":"ContainerDied","Data":"9ea4b632a5ff71cc4f0808f36e7e2e27cc30d597e6548a0b52edaa45718500f7"}
Sep 16 12:02:38.171168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:38.171157    2578 scope.go:117] "RemoveContainer" containerID="263f0f9e80d33475f91e5588f97c6b533fa9cb06ade5ef819c7e40f0a948ab36"
Sep 16 12:02:38.171422 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:38.171400    2578 scope.go:117] "RemoveContainer" containerID="9ea4b632a5ff71cc4f0808f36e7e2e27cc30d597e6548a0b52edaa45718500f7"
Sep 16 12:02:38.171625 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:38.171594    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8ls9j_openshift-console-operator(d2bd3fc1-6334-453c-8ca1-048c8b77bc17)\"" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" podUID="d2bd3fc1-6334-453c-8ca1-048c8b77bc17"
Sep 16 12:02:39.000170 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:39.000144    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kw862_4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f/dns-node-resolver/0.log"
Sep 16 12:02:39.176624 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:39.176594    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:02:39.177025 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:39.176914    2578 scope.go:117] "RemoveContainer" containerID="9ea4b632a5ff71cc4f0808f36e7e2e27cc30d597e6548a0b52edaa45718500f7"
Sep 16 12:02:39.177089 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:39.177071    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8ls9j_openshift-console-operator(d2bd3fc1-6334-453c-8ca1-048c8b77bc17)\"" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" podUID="d2bd3fc1-6334-453c-8ca1-048c8b77bc17"
Sep 16 12:02:39.179240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:39.179219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:39.179327 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:39.179313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:39.179389 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:39.179378    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:39.179439 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:39.179388    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:02:39.179439 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:39.179434    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:41.179420257 +0000 UTC m=+50.816745094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : configmap "kube-rbac-proxy" not found
Sep 16 12:02:39.179555 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:39.179447    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:41.179441414 +0000 UTC m=+50.816766230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:40.200465 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:40.200437    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wt7g7_ef67b5a6-8ba4-41b0-9f63-3a242a942d3c/node-ca/0.log"
Sep 16 12:02:40.803337 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:40.803311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qv7vb_9b44461a-fdcb-4a38-83d3-e72418354565/serve-healthcheck-canary/0.log"
Sep 16 12:02:41.198574 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:41.198474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:41.198730 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:41.198590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:41.198730 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:41.198630    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:41.198730 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:41.198695    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:45.198677397 +0000 UTC m=+54.836002224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:41.198730 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:41.198696    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:02:41.198887 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:41.198745    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:45.198732258 +0000 UTC m=+54.836057101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : configmap "kube-rbac-proxy" not found
Sep 16 12:02:42.107440 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:42.107397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:42.107440 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:42.107439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:42.107878 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.107571    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:42.107878 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.107637    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls podName:5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.107621744 +0000 UTC m=+67.744946559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rx5ch" (UID: "5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:02:42.107878 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.107572    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:42.107878 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.107674    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f9d96bb5-dj8r2: secret "image-registry-tls" not found
Sep 16 12:02:42.107878 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.107724    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls podName:1664da60-8154-46fe-9ce7-fd3656a0e422 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.107712702 +0000 UTC m=+67.745037519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls") pod "image-registry-6f9d96bb5-dj8r2" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422") : secret "image-registry-tls" not found
Sep 16 12:02:42.208758 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:42.208716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:42.208962 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:42.208849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:42.208962 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:42.208869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:42.208962 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.208880    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:02:42.208962 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:42.208888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:42.208962 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.208958    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls podName:228f1c38-5d40-47ec-9cf4-b11c93ce79aa nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.208931985 +0000 UTC m=+67.846256814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kmk6m" (UID: "228f1c38-5d40-47ec-9cf4-b11c93ce79aa") : secret "samples-operator-tls" not found
Sep 16 12:02:42.209216 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.208983    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:42.209216 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.208991    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:02:42.209216 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.209019    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls podName:acd2a478-2880-4d82-8a20-70e1e6a538c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.209009055 +0000 UTC m=+67.846333871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls") pod "dns-default-qz5zw" (UID: "acd2a478-2880-4d82-8a20-70e1e6a538c0") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:42.209216 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.209031    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.209025669 +0000 UTC m=+67.846350485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : secret "router-metrics-certs-default" not found
Sep 16 12:02:42.209216 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:42.209052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle podName:de22da40-9f25-4233-87ff-9fbc536d1a7e nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.209042612 +0000 UTC m=+67.846367428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle") pod "router-default-544f785cb6-6wc5m" (UID: "de22da40-9f25-4233-87ff-9fbc536d1a7e") : configmap references non-existent config key: service-ca.crt
Sep 16 12:02:45.237457 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:45.237417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:45.238029 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:45.237540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:45.238029 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:45.237588    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:45.238029 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:45.237658    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls podName:69ed17f4-800e-4a0b-b25d-d5e40307ee7a nodeName:}" failed. No retries permitted until 2026-09-16 12:02:53.237641354 +0000 UTC m=+62.874966183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8q8l" (UID: "69ed17f4-800e-4a0b-b25d-d5e40307ee7a") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:45.238140 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:45.238069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:46.720074 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:46.720037    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:46.720477 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:46.720104    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:02:46.720477 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:46.720434    2578 scope.go:117] "RemoveContainer" containerID="9ea4b632a5ff71cc4f0808f36e7e2e27cc30d597e6548a0b52edaa45718500f7"
Sep 16 12:02:46.720621 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:02:46.720602    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8ls9j_openshift-console-operator(d2bd3fc1-6334-453c-8ca1-048c8b77bc17)\"" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" podUID="d2bd3fc1-6334-453c-8ca1-048c8b77bc17"
Sep 16 12:02:52.071693 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:52.071653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kr5vm"
Sep 16 12:02:53.307305 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:53.307264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:53.309739 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:53.309718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ed17f4-800e-4a0b-b25d-d5e40307ee7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8q8l\" (UID: \"69ed17f4-800e-4a0b-b25d-d5e40307ee7a\") " pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:53.415955 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:53.415924    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4nl5\""
Sep 16 12:02:53.423457 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:53.423429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8q8l"
Sep 16 12:02:53.694727 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:53.694638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8q8l"]
Sep 16 12:02:53.698587 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:53.698551    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ed17f4_800e_4a0b_b25d_d5e40307ee7a.slice/crio-4d421941edefb4509ac6bd5e986908292c2741ccb5fd963beeb8b7b59d9a39d3 WatchSource:0}: Error finding container 4d421941edefb4509ac6bd5e986908292c2741ccb5fd963beeb8b7b59d9a39d3: Status 404 returned error can't find the container with id 4d421941edefb4509ac6bd5e986908292c2741ccb5fd963beeb8b7b59d9a39d3
Sep 16 12:02:54.228448 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:54.228408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8q8l" event={"ID":"69ed17f4-800e-4a0b-b25d-d5e40307ee7a","Type":"ContainerStarted","Data":"b26945605b7527d7ec13d9c7c8d539d41abcd5821916dd1921f7396c74bb291c"}
Sep 16 12:02:54.228647 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:54.228499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8q8l" event={"ID":"69ed17f4-800e-4a0b-b25d-d5e40307ee7a","Type":"ContainerStarted","Data":"4d421941edefb4509ac6bd5e986908292c2741ccb5fd963beeb8b7b59d9a39d3"}
Sep 16 12:02:55.234478 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:55.234439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8q8l" event={"ID":"69ed17f4-800e-4a0b-b25d-d5e40307ee7a","Type":"ContainerStarted","Data":"d33bced8c61761d93b7a48129b72756010a38e1edbca7ec8264ac143d6e785ad"}
Sep 16 12:02:56.241806 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.241774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8q8l" event={"ID":"69ed17f4-800e-4a0b-b25d-d5e40307ee7a","Type":"ContainerStarted","Data":"a99137b73fb74eadf79c4c21b78cecc6a9158b9a9b8120f4958c8f17fc3a9a30"}
Sep 16 12:02:56.261191 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.261142    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l8q8l" podStartSLOduration=17.159007265 podStartE2EDuration="19.261126907s" totalImagesPullingTime="2.102119642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:02:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:56.259239015 +0000 UTC m=+65.896563853" watchObservedRunningTime="2026-09-16 12:02:56.261126907 +0000 UTC m=+65.898451744"
Sep 16 12:02:56.538983 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.538878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:56.541771 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.541752    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:02:56.552095 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.552067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c58447-61b8-400e-be4b-8519523f5788-original-pull-secret\") pod \"global-pull-secret-syncer-8fst2\" (UID: \"04c58447-61b8-400e-be4b-8519523f5788\") " pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:56.639535 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.639482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:56.642525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.642488    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:02:56.651993 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.651970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16ec4baa-3a59-4ba5-afd3-eafb08071c5a-metrics-certs\") pod \"network-metrics-daemon-nhjbr\" (UID: \"16ec4baa-3a59-4ba5-afd3-eafb08071c5a\") " pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:56.691213 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.691185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fst2"
Sep 16 12:02:56.740347 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.740319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:56.743342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.743321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7ntk\" (UniqueName: \"kubernetes.io/projected/7a847c41-95f4-40fb-aaed-ee5f5057584d-kube-api-access-v7ntk\") pod \"network-check-target-j7tg4\" (UID: \"7a847c41-95f4-40fb-aaed-ee5f5057584d\") " pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:56.893471 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.893381    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qzcvp\""
Sep 16 12:02:56.898817 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.898793    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pcfgd\""
Sep 16 12:02:56.901820 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.901807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhjbr"
Sep 16 12:02:56.907238 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.907210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:56.960421 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:56.958744    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8fst2"]
Sep 16 12:02:56.963270 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:56.963239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c58447_61b8_400e_be4b_8519523f5788.slice/crio-6ec430f2d5f50d840949dda924dce74770867e1183afa2957d3d104581c7a84f WatchSource:0}: Error finding container 6ec430f2d5f50d840949dda924dce74770867e1183afa2957d3d104581c7a84f: Status 404 returned error can't find the container with id 6ec430f2d5f50d840949dda924dce74770867e1183afa2957d3d104581c7a84f
Sep 16 12:02:57.170063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:57.169971    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nhjbr"]
Sep 16 12:02:57.172556 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:57.172503    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ec4baa_3a59_4ba5_afd3_eafb08071c5a.slice/crio-7339c25f12d8b0ea7ff120e37adcdd4f50363e7a2f64fdf5c91fda5937a088d5 WatchSource:0}: Error finding container 7339c25f12d8b0ea7ff120e37adcdd4f50363e7a2f64fdf5c91fda5937a088d5: Status 404 returned error can't find the container with id 7339c25f12d8b0ea7ff120e37adcdd4f50363e7a2f64fdf5c91fda5937a088d5
Sep 16 12:02:57.191062 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:57.191033    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j7tg4"]
Sep 16 12:02:57.193750 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:57.193727    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a847c41_95f4_40fb_aaed_ee5f5057584d.slice/crio-a7d481dc1085634a2516481e2300c7c80dbd7a397a468376f076b13230e0369f WatchSource:0}: Error finding container a7d481dc1085634a2516481e2300c7c80dbd7a397a468376f076b13230e0369f: Status 404 returned error can't find the container with id a7d481dc1085634a2516481e2300c7c80dbd7a397a468376f076b13230e0369f
Sep 16 12:02:57.246874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:57.246845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j7tg4" event={"ID":"7a847c41-95f4-40fb-aaed-ee5f5057584d","Type":"ContainerStarted","Data":"a7d481dc1085634a2516481e2300c7c80dbd7a397a468376f076b13230e0369f"}
Sep 16 12:02:57.247228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:57.246986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:02:57.247947 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:57.247926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8fst2" event={"ID":"04c58447-61b8-400e-be4b-8519523f5788","Type":"ContainerStarted","Data":"6ec430f2d5f50d840949dda924dce74770867e1183afa2957d3d104581c7a84f"}
Sep 16 12:02:57.249122 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:57.249102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhjbr" event={"ID":"16ec4baa-3a59-4ba5-afd3-eafb08071c5a","Type":"ContainerStarted","Data":"7339c25f12d8b0ea7ff120e37adcdd4f50363e7a2f64fdf5c91fda5937a088d5"}
Sep 16 12:02:57.264231 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:57.264191    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j7tg4" podStartSLOduration=66.264179127 podStartE2EDuration="1m6.264179127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:57.26333395 +0000 UTC m=+66.900658789" watchObservedRunningTime="2026-09-16 12:02:57.264179127 +0000 UTC m=+66.901503961"
Sep 16 12:02:58.154371 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.154330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:58.154371 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.154374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:58.157656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.157625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"image-registry-6f9d96bb5-dj8r2\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") " pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:58.157656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.157643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rx5ch\" (UID: \"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:58.254814 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.254781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:58.254814 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.254819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:58.255344 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.254847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:58.255344 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.254884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:58.255534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.255485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j7tg4" event={"ID":"7a847c41-95f4-40fb-aaed-ee5f5057584d","Type":"ContainerStarted","Data":"effd498fdfcb1ebbb09a14c9f3ff8187e6f75de07a5ca4b129228b0c6dbe9e8e"}
Sep 16 12:02:58.256972 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.256906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de22da40-9f25-4233-87ff-9fbc536d1a7e-service-ca-bundle\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:58.258063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.258041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/228f1c38-5d40-47ec-9cf4-b11c93ce79aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kmk6m\" (UID: \"228f1c38-5d40-47ec-9cf4-b11c93ce79aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:58.258299 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.258279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de22da40-9f25-4233-87ff-9fbc536d1a7e-metrics-certs\") pod \"router-default-544f785cb6-6wc5m\" (UID: \"de22da40-9f25-4233-87ff-9fbc536d1a7e\") " pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:58.258653 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.258629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acd2a478-2880-4d82-8a20-70e1e6a538c0-metrics-tls\") pod \"dns-default-qz5zw\" (UID: \"acd2a478-2880-4d82-8a20-70e1e6a538c0\") " pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:58.365542 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.365498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mpvqx\""
Sep 16 12:02:58.373291 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.373270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:58.383422 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.383398    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gng5k\""
Sep 16 12:02:58.391163 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.391144    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"
Sep 16 12:02:58.425551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.425242    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4c4mq\""
Sep 16 12:02:58.432656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.432595    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"
Sep 16 12:02:58.442035 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.441804    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8hlmq\""
Sep 16 12:02:58.450565 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.449835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:02:58.462150 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.461925    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lxplz\""
Sep 16 12:02:58.474063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.469808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:58.666861 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.666840    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f9d96bb5-dj8r2"]
Sep 16 12:02:58.668667 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:58.668640    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1664da60_8154_46fe_9ce7_fd3656a0e422.slice/crio-593d93b60a875e35a136c09985b43d849e127f4d98b91f138aaf5ac2870ddee3 WatchSource:0}: Error finding container 593d93b60a875e35a136c09985b43d849e127f4d98b91f138aaf5ac2870ddee3: Status 404 returned error can't find the container with id 593d93b60a875e35a136c09985b43d849e127f4d98b91f138aaf5ac2870ddee3
Sep 16 12:02:58.688804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.688195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch"]
Sep 16 12:02:58.695481 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:58.695450    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad945a4_0cf4_4e6b_99e8_8a21ce8323c9.slice/crio-a4da4afa4a3f8200044774b4a45bd84e137c41a2ab2d7ee5556b2c25ee04c57c WatchSource:0}: Error finding container a4da4afa4a3f8200044774b4a45bd84e137c41a2ab2d7ee5556b2c25ee04c57c: Status 404 returned error can't find the container with id a4da4afa4a3f8200044774b4a45bd84e137c41a2ab2d7ee5556b2c25ee04c57c
Sep 16 12:02:58.745293 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.745271    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m"]
Sep 16 12:02:58.773915 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.773842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qz5zw"]
Sep 16 12:02:58.777464 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:58.777436    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd2a478_2880_4d82_8a20_70e1e6a538c0.slice/crio-0aeda1c65b2886eb9d419e265a9667fbedddbedf21018fd7eaa4332c2f07b506 WatchSource:0}: Error finding container 0aeda1c65b2886eb9d419e265a9667fbedddbedf21018fd7eaa4332c2f07b506: Status 404 returned error can't find the container with id 0aeda1c65b2886eb9d419e265a9667fbedddbedf21018fd7eaa4332c2f07b506
Sep 16 12:02:58.783340 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:58.783272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-544f785cb6-6wc5m"]
Sep 16 12:02:58.786149 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:02:58.786073    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde22da40_9f25_4233_87ff_9fbc536d1a7e.slice/crio-df3f07577518e0efbd21cfb79b89715259faa322fbd555f6affa3de1598d6320 WatchSource:0}: Error finding container df3f07577518e0efbd21cfb79b89715259faa322fbd555f6affa3de1598d6320: Status 404 returned error can't find the container with id df3f07577518e0efbd21cfb79b89715259faa322fbd555f6affa3de1598d6320
Sep 16 12:02:59.262175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.262138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhjbr" event={"ID":"16ec4baa-3a59-4ba5-afd3-eafb08071c5a","Type":"ContainerStarted","Data":"9720c5a362ec75e19b588288a7a2c74b7e9bcd5ccfa751bc04021b30eff970a1"}
Sep 16 12:02:59.262668 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.262201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhjbr" event={"ID":"16ec4baa-3a59-4ba5-afd3-eafb08071c5a","Type":"ContainerStarted","Data":"1e84a240cce2abb48f12ce792c460a68e656d3a4cc3c7cb82f53abbdf3eb38b9"}
Sep 16 12:02:59.264297 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.264256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-544f785cb6-6wc5m" event={"ID":"de22da40-9f25-4233-87ff-9fbc536d1a7e","Type":"ContainerStarted","Data":"06da092f04530a0b14fb7c3f967c2928c85eaf142aa29413f48f5e01943aea26"}
Sep 16 12:02:59.264406 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.264309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-544f785cb6-6wc5m" event={"ID":"de22da40-9f25-4233-87ff-9fbc536d1a7e","Type":"ContainerStarted","Data":"df3f07577518e0efbd21cfb79b89715259faa322fbd555f6affa3de1598d6320"}
Sep 16 12:02:59.265547 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.265502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qz5zw" event={"ID":"acd2a478-2880-4d82-8a20-70e1e6a538c0","Type":"ContainerStarted","Data":"0aeda1c65b2886eb9d419e265a9667fbedddbedf21018fd7eaa4332c2f07b506"}
Sep 16 12:02:59.266830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.266800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m" event={"ID":"228f1c38-5d40-47ec-9cf4-b11c93ce79aa","Type":"ContainerStarted","Data":"c9ac0b8886bbb1323cb214b3481c80c45b728a186f4a871359370caf39053ff3"}
Sep 16 12:02:59.268218 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.268194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch" event={"ID":"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9","Type":"ContainerStarted","Data":"a4da4afa4a3f8200044774b4a45bd84e137c41a2ab2d7ee5556b2c25ee04c57c"}
Sep 16 12:02:59.269915 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.269894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2" event={"ID":"1664da60-8154-46fe-9ce7-fd3656a0e422","Type":"ContainerStarted","Data":"4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a"}
Sep 16 12:02:59.270085 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.270066    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:02:59.270166 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.270113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2" event={"ID":"1664da60-8154-46fe-9ce7-fd3656a0e422","Type":"ContainerStarted","Data":"593d93b60a875e35a136c09985b43d849e127f4d98b91f138aaf5ac2870ddee3"}
Sep 16 12:02:59.282495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.282436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nhjbr" podStartSLOduration=67.159183709 podStartE2EDuration="1m8.282416311s" totalImagesPullingTime="1.123232602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:59.280724686 +0000 UTC m=+68.918049524" watchObservedRunningTime="2026-09-16 12:02:59.282416311 +0000 UTC m=+68.919741151"
Sep 16 12:02:59.306319 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.306252    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2" podStartSLOduration=67.306232728 podStartE2EDuration="1m7.306232728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:59.305559626 +0000 UTC m=+68.942884464" watchObservedRunningTime="2026-09-16 12:02:59.306232728 +0000 UTC m=+68.943557566"
Sep 16 12:02:59.328876 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.328821    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-544f785cb6-6wc5m" podStartSLOduration=56.32880365 podStartE2EDuration="56.32880365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:59.326969496 +0000 UTC m=+68.964294334" watchObservedRunningTime="2026-09-16 12:02:59.32880365 +0000 UTC m=+68.966128482"
Sep 16 12:02:59.471078 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.471037    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:59.474413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.474328    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:02:59.878940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:02:59.878822    2578 scope.go:117] "RemoveContainer" containerID="9ea4b632a5ff71cc4f0808f36e7e2e27cc30d597e6548a0b52edaa45718500f7"
Sep 16 12:03:00.081844 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.081772    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"]
Sep 16 12:03:00.086930 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.086325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.089189 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.089142    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:03:00.089600 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.089581    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 12:03:00.089825 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.089808    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:03:00.090457 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.090422    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:03:00.101669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.098298    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"]
Sep 16 12:03:00.179094 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.179013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17aa1082-8133-46e5-9322-491775c34e40-tmp\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.179094 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.179070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/17aa1082-8133-46e5-9322-491775c34e40-klusterlet-config\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.179321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.179098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbxcl\" (UniqueName: \"kubernetes.io/projected/17aa1082-8133-46e5-9322-491775c34e40-kube-api-access-sbxcl\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.287186 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.287141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17aa1082-8133-46e5-9322-491775c34e40-tmp\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.288358 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.287204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/17aa1082-8133-46e5-9322-491775c34e40-klusterlet-config\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.288358 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.287231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbxcl\" (UniqueName: \"kubernetes.io/projected/17aa1082-8133-46e5-9322-491775c34e40-kube-api-access-sbxcl\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.288358 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.288270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17aa1082-8133-46e5-9322-491775c34e40-tmp\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.288725 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.288704    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:03:00.288980 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.288836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" event={"ID":"d2bd3fc1-6334-453c-8ca1-048c8b77bc17","Type":"ContainerStarted","Data":"f19f466f52cee7f4a04f1b897f18e8644bbe13dd5ed224d337ac60dec3a079f3"}
Sep 16 12:03:00.290107 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.290035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:03:00.290107 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.290073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:03:00.292116 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.292055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-544f785cb6-6wc5m"
Sep 16 12:03:00.293269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.293221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/17aa1082-8133-46e5-9322-491775c34e40-klusterlet-config\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.303300 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.303242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbxcl\" (UniqueName: \"kubernetes.io/projected/17aa1082-8133-46e5-9322-491775c34e40-kube-api-access-sbxcl\") pod \"klusterlet-addon-workmgr-688cfb5496-4xc6k\" (UID: \"17aa1082-8133-46e5-9322-491775c34e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:00.319434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.319249    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" podStartSLOduration=49.463650177 podStartE2EDuration="57.319224512s" totalImagesPullingTime="7.855574335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:00.317809386 +0000 UTC m=+69.955134223" watchObservedRunningTime="2026-09-16 12:03:00.319224512 +0000 UTC m=+69.956549351"
Sep 16 12:03:00.413284 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:00.413239    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:01.290014 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.289966    2578 patch_prober.go:28] interesting pod/console-operator-696884fb9c-8ls9j container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:03:01.290420 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.290046    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j" podUID="d2bd3fc1-6334-453c-8ca1-048c8b77bc17" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:03:01.471778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.471746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-8ls9j"
Sep 16 12:03:01.657067 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.657028    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-t8qqw"]
Sep 16 12:03:01.680677 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.680646    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-t8qqw"]
Sep 16 12:03:01.680841 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.680775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-t8qqw"
Sep 16 12:03:01.686622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.686591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9btz7\""
Sep 16 12:03:01.686785 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.686632    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:03:01.686785 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.686592    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:03:01.806718 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.806618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhcs\" (UniqueName: \"kubernetes.io/projected/a35c46ce-8537-451f-97c6-f31e09906543-kube-api-access-nnhcs\") pod \"downloads-76f99584f4-t8qqw\" (UID: \"a35c46ce-8537-451f-97c6-f31e09906543\") " pod="openshift-console/downloads-76f99584f4-t8qqw"
Sep 16 12:03:01.907230 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.907194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhcs\" (UniqueName: \"kubernetes.io/projected/a35c46ce-8537-451f-97c6-f31e09906543-kube-api-access-nnhcs\") pod \"downloads-76f99584f4-t8qqw\" (UID: \"a35c46ce-8537-451f-97c6-f31e09906543\") " pod="openshift-console/downloads-76f99584f4-t8qqw"
Sep 16 12:03:01.916986 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.916946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhcs\" (UniqueName: \"kubernetes.io/projected/a35c46ce-8537-451f-97c6-f31e09906543-kube-api-access-nnhcs\") pod \"downloads-76f99584f4-t8qqw\" (UID: \"a35c46ce-8537-451f-97c6-f31e09906543\") " pod="openshift-console/downloads-76f99584f4-t8qqw"
Sep 16 12:03:01.993386 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:01.993349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-t8qqw"
Sep 16 12:03:04.499231 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:04.499118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"]
Sep 16 12:03:04.500321 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:03:04.500268    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17aa1082_8133_46e5_9322_491775c34e40.slice/crio-b5810ec680295603f058d4828e03fe2c3581b947921c7439361888737e97a7ff WatchSource:0}: Error finding container b5810ec680295603f058d4828e03fe2c3581b947921c7439361888737e97a7ff: Status 404 returned error can't find the container with id b5810ec680295603f058d4828e03fe2c3581b947921c7439361888737e97a7ff
Sep 16 12:03:04.511990 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:04.511961    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-t8qqw"]
Sep 16 12:03:04.850379 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:04.850287    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"]
Sep 16 12:03:04.867766 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:04.867738    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"]
Sep 16 12:03:04.867920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:04.867858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"
Sep 16 12:03:04.871180 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:04.871114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 12:03:04.871180 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:04.871167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4l8hq\""
Sep 16 12:03:05.030863 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.030827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hjqqz\" (UID: \"9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"
Sep 16 12:03:05.132145 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.132065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hjqqz\" (UID: \"9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"
Sep 16 12:03:05.134589 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.134570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hjqqz\" (UID: \"9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"
Sep 16 12:03:05.178639 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.178610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"
Sep 16 12:03:05.324242 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.324208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m" event={"ID":"228f1c38-5d40-47ec-9cf4-b11c93ce79aa","Type":"ContainerStarted","Data":"c3f2e229e82ac6a75ff8bf942668f736da410fe519fa2135c9e4ce91782f0e4a"}
Sep 16 12:03:05.324395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.324276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m" event={"ID":"228f1c38-5d40-47ec-9cf4-b11c93ce79aa","Type":"ContainerStarted","Data":"2552af35afa7348e9f2b5f390ac193e9fe04a2b1f760ed81fa14f37c3d743e05"}
Sep 16 12:03:05.325862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.325831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8fst2" event={"ID":"04c58447-61b8-400e-be4b-8519523f5788","Type":"ContainerStarted","Data":"f6cd2931a12dd70df73fe29860b6e0cfdccdd45105f0094006d060c5f2f3cf0d"}
Sep 16 12:03:05.327302 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.327280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch" event={"ID":"5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9","Type":"ContainerStarted","Data":"8c3cd5fcc2dbdc3582c590f6d437b93ff2e7b985b52774124c26e1e263160c01"}
Sep 16 12:03:05.328422 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.328396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-t8qqw" event={"ID":"a35c46ce-8537-451f-97c6-f31e09906543","Type":"ContainerStarted","Data":"60a4de9da347714551e6c726ae3b27f3a9d8414445d4dd89f41c02059cbaf3a4"}
Sep 16 12:03:05.329548 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.329525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k" event={"ID":"17aa1082-8133-46e5-9322-491775c34e40","Type":"ContainerStarted","Data":"b5810ec680295603f058d4828e03fe2c3581b947921c7439361888737e97a7ff"}
Sep 16 12:03:05.331103 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.331081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qz5zw" event={"ID":"acd2a478-2880-4d82-8a20-70e1e6a538c0","Type":"ContainerStarted","Data":"2c5619e7c869969d3f46f344fad6ad41d9c271054aedf576b683be3ce146b675"}
Sep 16 12:03:05.331187 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.331127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qz5zw" event={"ID":"acd2a478-2880-4d82-8a20-70e1e6a538c0","Type":"ContainerStarted","Data":"c5cc2854582beb7f52305622a05e2f4d4984392dec8346f27dbe4d0c63ed1517"}
Sep 16 12:03:05.331232 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.331199    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:03:05.341445 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.341409    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kmk6m" podStartSLOduration=56.905071058 podStartE2EDuration="1m2.341396014s" totalImagesPullingTime="5.436324956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:05.341016517 +0000 UTC m=+74.978341341" watchObservedRunningTime="2026-09-16 12:03:05.341396014 +0000 UTC m=+74.978720845"
Sep 16 12:03:05.359067 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.358973    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qz5zw" podStartSLOduration=33.909048913 podStartE2EDuration="39.358959415s" totalImagesPullingTime="5.449910502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:05.358877598 +0000 UTC m=+74.996202449" watchObservedRunningTime="2026-09-16 12:03:05.358959415 +0000 UTC m=+74.996284256"
Sep 16 12:03:05.377840 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.377794    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8fst2" podStartSLOduration=67.119212549 podStartE2EDuration="1m14.377773646s" totalImagesPullingTime="7.258561097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:05.376684046 +0000 UTC m=+75.014008885" watchObservedRunningTime="2026-09-16 12:03:05.377773646 +0000 UTC m=+75.015098488"
Sep 16 12:03:05.399130 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.399047    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rx5ch" podStartSLOduration=56.871121546 podStartE2EDuration="1m2.39903077s" totalImagesPullingTime="5.527909224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:05.397846086 +0000 UTC m=+75.035170925" watchObservedRunningTime="2026-09-16 12:03:05.39903077 +0000 UTC m=+75.036355608"
Sep 16 12:03:05.469750 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:05.469634    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"]
Sep 16 12:03:05.472736 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:03:05.472702    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fefc4a2_e8f1_4a3d_b114_78c1cc1d6779.slice/crio-916cfc757718df31d9735f02d1909e84eea3d3e361d802f2a3191cfcf617bd2d WatchSource:0}: Error finding container 916cfc757718df31d9735f02d1909e84eea3d3e361d802f2a3191cfcf617bd2d: Status 404 returned error can't find the container with id 916cfc757718df31d9735f02d1909e84eea3d3e361d802f2a3191cfcf617bd2d
Sep 16 12:03:06.338727 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:06.338691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz" event={"ID":"9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779","Type":"ContainerStarted","Data":"916cfc757718df31d9735f02d1909e84eea3d3e361d802f2a3191cfcf617bd2d"}
Sep 16 12:03:10.357949 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:10.357912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k" event={"ID":"17aa1082-8133-46e5-9322-491775c34e40","Type":"ContainerStarted","Data":"f02e9a240b283a9739fbbbeb64e373d4277c7331ec34f5a79f313e72f3ea9e80"}
Sep 16 12:03:10.358460 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:10.358082    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:10.360257 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:10.360232    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k"
Sep 16 12:03:10.360409 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:10.360276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz" event={"ID":"9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779","Type":"ContainerStarted","Data":"19897d21fac802e836e826a2b0fb1b86ace016f8d1e4e9c4b5b1018d299f8d31"}
Sep 16 12:03:10.360478 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:10.360436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"
Sep 16 12:03:10.365697 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:10.365675    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz"
Sep 16 12:03:10.375091 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:10.375045    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-688cfb5496-4xc6k" podStartSLOduration=5.197875816 podStartE2EDuration="10.375029319s" totalImagesPullingTime="5.177153503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:10.373731095 +0000 UTC m=+80.011055933" watchObservedRunningTime="2026-09-16 12:03:10.375029319 +0000 UTC m=+80.012354157"
Sep 16 12:03:10.406782 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:10.406719    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hjqqz" podStartSLOduration=2.222305419 podStartE2EDuration="6.406695239s" totalImagesPullingTime="4.18438982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:10.405834353 +0000 UTC m=+80.043159191" watchObservedRunningTime="2026-09-16 12:03:10.406695239 +0000 UTC m=+80.044020078"
Sep 16 12:03:15.310827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.310688    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c8tz2"]
Sep 16 12:03:15.313475 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.313454    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.316098 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.316073    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:03:15.316208 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.316154    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:03:15.316208 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.316155    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:03:15.316461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.316444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-twxkx\""
Sep 16 12:03:15.316556 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.316451    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:03:15.320349 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.320349 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.320349 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-wtmp\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.320349 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5zxf\" (UniqueName: \"kubernetes.io/projected/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-kube-api-access-k5zxf\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.320349 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-root\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.320667 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-sys\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.320667 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-metrics-client-ca\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.320667 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-textfile\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.320667 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.320533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-tls\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.340975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.340951    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qz5zw"
Sep 16 12:03:15.421816 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.421780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5zxf\" (UniqueName: \"kubernetes.io/projected/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-kube-api-access-k5zxf\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.421991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.421836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-root\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.421991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.421915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-root\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422119 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.421994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-sys\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422119 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-metrics-client-ca\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422119 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-sys\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422267 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-textfile\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422267 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-tls\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422267 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422414 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422414 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-wtmp\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422535 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-wtmp\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422738 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-metrics-client-ca\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.422878 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.422863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-textfile\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.423289 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.423259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.425053 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.425010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-tls\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.425266 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.425238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.432182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.432150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5zxf\" (UniqueName: \"kubernetes.io/projected/4fd22cc8-bcd2-489e-b1e4-591b8078ca14-kube-api-access-k5zxf\") pod \"node-exporter-c8tz2\" (UID: \"4fd22cc8-bcd2-489e-b1e4-591b8078ca14\") " pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:15.624869 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:15.624777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8tz2"
Sep 16 12:03:18.378801 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:18.378759    2578 patch_prober.go:28] interesting pod/image-registry-6f9d96bb5-dj8r2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 12:03:18.379241 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:18.378825    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2" podUID="1664da60-8154-46fe-9ce7-fd3656a0e422" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:03:20.294825 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:20.294792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:03:22.378865 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:03:22.378835    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd22cc8_bcd2_489e_b1e4_591b8078ca14.slice/crio-8832c02b061e022ba5f87ca843cc06be49133bc3eeeeeb26bd6fd269d8ced8d8 WatchSource:0}: Error finding container 8832c02b061e022ba5f87ca843cc06be49133bc3eeeeeb26bd6fd269d8ced8d8: Status 404 returned error can't find the container with id 8832c02b061e022ba5f87ca843cc06be49133bc3eeeeeb26bd6fd269d8ced8d8
Sep 16 12:03:22.417149 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:22.417113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8tz2" event={"ID":"4fd22cc8-bcd2-489e-b1e4-591b8078ca14","Type":"ContainerStarted","Data":"8832c02b061e022ba5f87ca843cc06be49133bc3eeeeeb26bd6fd269d8ced8d8"}
Sep 16 12:03:23.424941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:23.424894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-t8qqw" event={"ID":"a35c46ce-8537-451f-97c6-f31e09906543","Type":"ContainerStarted","Data":"778b4504fae75dcf32c6ce303e16190d8ae5c406ff32c9cb782a53bf77d7133a"}
Sep 16 12:03:23.426102 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:23.426051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-t8qqw"
Sep 16 12:03:23.441107 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:23.441067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-t8qqw"
Sep 16 12:03:23.447131 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:23.446805    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-t8qqw" podStartSLOduration=4.207806193 podStartE2EDuration="22.44678763s" totalImagesPullingTime="18.238981437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:23.444589496 +0000 UTC m=+93.081914337" watchObservedRunningTime="2026-09-16 12:03:23.44678763 +0000 UTC m=+93.084112467"
Sep 16 12:03:23.518416 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:23.518386    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f9d96bb5-dj8r2"]
Sep 16 12:03:24.431834 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:24.431801    2578 generic.go:356] "Generic (PLEG): container finished" podID="4fd22cc8-bcd2-489e-b1e4-591b8078ca14" containerID="97bd41b43450e29a2e1f0c70a5bba8b1afb16baf28c44442a61cca65e9a5e9d0" exitCode=0
Sep 16 12:03:24.432316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:24.431890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8tz2" event={"ID":"4fd22cc8-bcd2-489e-b1e4-591b8078ca14","Type":"ContainerDied","Data":"97bd41b43450e29a2e1f0c70a5bba8b1afb16baf28c44442a61cca65e9a5e9d0"}
Sep 16 12:03:25.438995 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:25.438958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8tz2" event={"ID":"4fd22cc8-bcd2-489e-b1e4-591b8078ca14","Type":"ContainerStarted","Data":"3d55c1f83a7c833163a71b72fc7d34915dcdcc8e475029eb6fe7f3dfcb20dbda"}
Sep 16 12:03:25.439565 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:25.439016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8tz2" event={"ID":"4fd22cc8-bcd2-489e-b1e4-591b8078ca14","Type":"ContainerStarted","Data":"b1785f7c4f33d84b5e40e8a33fd22bcee1c7ba57593d9f4aa53c7834f17fb156"}
Sep 16 12:03:25.461467 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:25.461400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c8tz2" podStartSLOduration=9.245836008 podStartE2EDuration="10.461382625s" totalImagesPullingTime="1.128973256s" totalInitContainerRuntime="86.573361ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:25.460206106 +0000 UTC m=+95.097530955" watchObservedRunningTime="2026-09-16 12:03:25.461382625 +0000 UTC m=+95.098707465"
Sep 16 12:03:29.273144 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:29.273097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j7tg4"
Sep 16 12:03:31.768137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.768100    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5456c78c44-rn7bl"]
Sep 16 12:03:31.822495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.822457    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5456c78c44-rn7bl"]
Sep 16 12:03:31.822707 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.822618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.826064 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.826040    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:03:31.826276 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.826264    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:03:31.826689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.826664    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:03:31.826797 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.826731    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:03:31.827811 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.827780    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:03:31.827811 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.827807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jg44n\""
Sep 16 12:03:31.832059 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.831937    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:03:31.865166 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.865110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-console-config\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.865166 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.865163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-oauth-serving-cert\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.865430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.865183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgd6\" (UniqueName: \"kubernetes.io/projected/33897d11-6d61-4156-b659-a431703acf91-kube-api-access-8zgd6\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.865430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.865276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-trusted-ca-bundle\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.865430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.865342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-serving-cert\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.865430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.865371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-oauth-config\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.865430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.865421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-service-ca\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.966850 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.966798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-service-ca\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.967063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.966893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-console-config\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.967063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.966961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-oauth-serving-cert\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.967063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.966987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgd6\" (UniqueName: \"kubernetes.io/projected/33897d11-6d61-4156-b659-a431703acf91-kube-api-access-8zgd6\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.967228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.967103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-trusted-ca-bundle\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.967228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.967157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-serving-cert\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.967228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.967186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-oauth-config\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.967684 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.967655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-service-ca\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.967791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.967748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-oauth-serving-cert\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.968057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.968032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-console-config\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.968623 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.968595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-trusted-ca-bundle\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.969991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.969957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-oauth-config\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.970285 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.970266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-serving-cert\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:31.975379 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:31.975356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgd6\" (UniqueName: \"kubernetes.io/projected/33897d11-6d61-4156-b659-a431703acf91-kube-api-access-8zgd6\") pod \"console-5456c78c44-rn7bl\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") " pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:32.134415 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:32.134323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:32.431721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:32.431695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5456c78c44-rn7bl"]
Sep 16 12:03:32.434311 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:03:32.434285    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33897d11_6d61_4156_b659_a431703acf91.slice/crio-3f7832c590cac441b7272d303cc653dbedd0d0b7f54a94963b58714fca5ff19e WatchSource:0}: Error finding container 3f7832c590cac441b7272d303cc653dbedd0d0b7f54a94963b58714fca5ff19e: Status 404 returned error can't find the container with id 3f7832c590cac441b7272d303cc653dbedd0d0b7f54a94963b58714fca5ff19e
Sep 16 12:03:32.471150 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:32.471120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5456c78c44-rn7bl" event={"ID":"33897d11-6d61-4156-b659-a431703acf91","Type":"ContainerStarted","Data":"3f7832c590cac441b7272d303cc653dbedd0d0b7f54a94963b58714fca5ff19e"}
Sep 16 12:03:36.488842 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:36.488805    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5456c78c44-rn7bl" event={"ID":"33897d11-6d61-4156-b659-a431703acf91","Type":"ContainerStarted","Data":"46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1"}
Sep 16 12:03:36.506329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:36.506286    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5456c78c44-rn7bl" podStartSLOduration=2.039239172 podStartE2EDuration="5.50627187s" totalImagesPullingTime="3.467032698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:36.505221309 +0000 UTC m=+106.142546149" watchObservedRunningTime="2026-09-16 12:03:36.50627187 +0000 UTC m=+106.143596707"
Sep 16 12:03:42.134693 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:42.134643    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:42.135291 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:42.134726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:42.139503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:42.139478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:42.514387 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:42.514359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:03:47.531287 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:47.531250    2578 generic.go:356] "Generic (PLEG): container finished" podID="fdeaddaa-5009-4d0b-99b6-57f6bb8b219a" containerID="481532cfdf9f3d3f71270a0248a4841520f801ce9ed91e9d91982e3b02e8c729" exitCode=0
Sep 16 12:03:47.531680 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:47.531326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6" event={"ID":"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a","Type":"ContainerDied","Data":"481532cfdf9f3d3f71270a0248a4841520f801ce9ed91e9d91982e3b02e8c729"}
Sep 16 12:03:47.531680 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:47.531640    2578 scope.go:117] "RemoveContainer" containerID="481532cfdf9f3d3f71270a0248a4841520f801ce9ed91e9d91982e3b02e8c729"
Sep 16 12:03:48.537009 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:48.536980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jsvk6" event={"ID":"fdeaddaa-5009-4d0b-99b6-57f6bb8b219a","Type":"ContainerStarted","Data":"5754d3e7ba92532be27c78524225779e2376b5a6f88ed53ef3dffa0be8fe6602"}
Sep 16 12:03:48.556353 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:48.556317    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2" podUID="1664da60-8154-46fe-9ce7-fd3656a0e422" containerName="registry" containerID="cri-o://4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a" gracePeriod=30
Sep 16 12:03:48.836850 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:48.836811    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:03:49.011976 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.011939    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-image-registry-private-configuration\") pod \"1664da60-8154-46fe-9ce7-fd3656a0e422\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") "
Sep 16 12:03:49.012168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.011985    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-certificates\") pod \"1664da60-8154-46fe-9ce7-fd3656a0e422\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") "
Sep 16 12:03:49.012168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.012022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfm6f\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-kube-api-access-hfm6f\") pod \"1664da60-8154-46fe-9ce7-fd3656a0e422\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") "
Sep 16 12:03:49.012168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.012052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-trusted-ca\") pod \"1664da60-8154-46fe-9ce7-fd3656a0e422\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") "
Sep 16 12:03:49.012168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.012070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-bound-sa-token\") pod \"1664da60-8154-46fe-9ce7-fd3656a0e422\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") "
Sep 16 12:03:49.012168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.012102    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") pod \"1664da60-8154-46fe-9ce7-fd3656a0e422\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") "
Sep 16 12:03:49.012168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.012127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1664da60-8154-46fe-9ce7-fd3656a0e422-ca-trust-extracted\") pod \"1664da60-8154-46fe-9ce7-fd3656a0e422\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") "
Sep 16 12:03:49.012474 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.012174    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-installation-pull-secrets\") pod \"1664da60-8154-46fe-9ce7-fd3656a0e422\" (UID: \"1664da60-8154-46fe-9ce7-fd3656a0e422\") "
Sep 16 12:03:49.012474 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.012432    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1664da60-8154-46fe-9ce7-fd3656a0e422" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:03:49.013508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.012973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1664da60-8154-46fe-9ce7-fd3656a0e422" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:03:49.015119 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.015090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1664da60-8154-46fe-9ce7-fd3656a0e422" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:03:49.015264 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.015231    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-kube-api-access-hfm6f" (OuterVolumeSpecName: "kube-api-access-hfm6f") pod "1664da60-8154-46fe-9ce7-fd3656a0e422" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422"). InnerVolumeSpecName "kube-api-access-hfm6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:03:49.015324 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.015264    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1664da60-8154-46fe-9ce7-fd3656a0e422" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:03:49.015361 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.015320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1664da60-8154-46fe-9ce7-fd3656a0e422" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:03:49.015808 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.015791    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1664da60-8154-46fe-9ce7-fd3656a0e422" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:03:49.021263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.021239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1664da60-8154-46fe-9ce7-fd3656a0e422-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1664da60-8154-46fe-9ce7-fd3656a0e422" (UID: "1664da60-8154-46fe-9ce7-fd3656a0e422"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:03:49.112958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.112870    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-installation-pull-secrets\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:03:49.112958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.112903    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1664da60-8154-46fe-9ce7-fd3656a0e422-image-registry-private-configuration\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:03:49.112958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.112913    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-certificates\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:03:49.112958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.112921    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfm6f\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-kube-api-access-hfm6f\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:03:49.112958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.112930    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1664da60-8154-46fe-9ce7-fd3656a0e422-trusted-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:03:49.112958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.112938    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-bound-sa-token\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:03:49.112958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.112945    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1664da60-8154-46fe-9ce7-fd3656a0e422-registry-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:03:49.112958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.112953    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1664da60-8154-46fe-9ce7-fd3656a0e422-ca-trust-extracted\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:03:49.542384 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.542349    2578 generic.go:356] "Generic (PLEG): container finished" podID="1664da60-8154-46fe-9ce7-fd3656a0e422" containerID="4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a" exitCode=0
Sep 16 12:03:49.542886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.542404    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2"
Sep 16 12:03:49.542886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.542443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2" event={"ID":"1664da60-8154-46fe-9ce7-fd3656a0e422","Type":"ContainerDied","Data":"4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a"}
Sep 16 12:03:49.542886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.542498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9d96bb5-dj8r2" event={"ID":"1664da60-8154-46fe-9ce7-fd3656a0e422","Type":"ContainerDied","Data":"593d93b60a875e35a136c09985b43d849e127f4d98b91f138aaf5ac2870ddee3"}
Sep 16 12:03:49.542886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.542545    2578 scope.go:117] "RemoveContainer" containerID="4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a"
Sep 16 12:03:49.555586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.555566    2578 scope.go:117] "RemoveContainer" containerID="4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a"
Sep 16 12:03:49.555881 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:03:49.555863    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a\": container with ID starting with 4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a not found: ID does not exist" containerID="4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a"
Sep 16 12:03:49.555948 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.555890    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a"} err="failed to get container status \"4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a\": rpc error: code = NotFound desc = could not find container \"4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a\": container with ID starting with 4a3c2c92b1741b388e53c928271efec0a623a2d3e9000385fd835c8b8913ca2a not found: ID does not exist"
Sep 16 12:03:49.563299 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.563272    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f9d96bb5-dj8r2"]
Sep 16 12:03:49.566642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:49.566617    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f9d96bb5-dj8r2"]
Sep 16 12:03:50.882656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:50.882626    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1664da60-8154-46fe-9ce7-fd3656a0e422" path="/var/lib/kubelet/pods/1664da60-8154-46fe-9ce7-fd3656a0e422/volumes"
Sep 16 12:03:52.555758 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:52.555727    2578 generic.go:356] "Generic (PLEG): container finished" podID="14f56708-c1b2-4ac8-8a6b-3eba48cfb62c" containerID="3e9219d504afeafa746891ff0053403639d135d91a0776ceb79ebd8293febf38" exitCode=0
Sep 16 12:03:52.556144 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:52.555790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-v7sp6" event={"ID":"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c","Type":"ContainerDied","Data":"3e9219d504afeafa746891ff0053403639d135d91a0776ceb79ebd8293febf38"}
Sep 16 12:03:52.556144 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:52.556102    2578 scope.go:117] "RemoveContainer" containerID="3e9219d504afeafa746891ff0053403639d135d91a0776ceb79ebd8293febf38"
Sep 16 12:03:53.563038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:03:53.562998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-v7sp6" event={"ID":"14f56708-c1b2-4ac8-8a6b-3eba48cfb62c","Type":"ContainerStarted","Data":"0d4ca58b5044f0f7c85f7b91c66e2f8fc9948368224167a0e657961ee513fe3c"}
Sep 16 12:04:07.616404 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:07.616368    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b3e46af-da21-45fc-8b38-f6754a2a7c70" containerID="0ad98db108ff43f4c737f1d7bbddf258df96bc72de3c13106f5bb1967de5da39" exitCode=0
Sep 16 12:04:07.616791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:07.616437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx" event={"ID":"1b3e46af-da21-45fc-8b38-f6754a2a7c70","Type":"ContainerDied","Data":"0ad98db108ff43f4c737f1d7bbddf258df96bc72de3c13106f5bb1967de5da39"}
Sep 16 12:04:07.616791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:07.616748    2578 scope.go:117] "RemoveContainer" containerID="0ad98db108ff43f4c737f1d7bbddf258df96bc72de3c13106f5bb1967de5da39"
Sep 16 12:04:08.623082 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:08.623045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x8hnx" event={"ID":"1b3e46af-da21-45fc-8b38-f6754a2a7c70","Type":"ContainerStarted","Data":"1c185392fd1170202e9b5af2cc7bddd8f2440b2b76e117dbd351d66ce0494337"}
Sep 16 12:04:37.295128 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.295088    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76c8c8844d-t5xsw"]
Sep 16 12:04:37.295614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.295398    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1664da60-8154-46fe-9ce7-fd3656a0e422" containerName="registry"
Sep 16 12:04:37.295614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.295412    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1664da60-8154-46fe-9ce7-fd3656a0e422" containerName="registry"
Sep 16 12:04:37.295614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.295469    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1664da60-8154-46fe-9ce7-fd3656a0e422" containerName="registry"
Sep 16 12:04:37.298014 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.297994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.312161 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.312134    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c8c8844d-t5xsw"]
Sep 16 12:04:37.377440 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.377408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-oauth-serving-cert\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.377596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.377446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ck8g\" (UniqueName: \"kubernetes.io/projected/016d9fb9-4668-4a37-bb01-9871dc33f6ce-kube-api-access-5ck8g\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.377596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.377474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-service-ca\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.377596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.377555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-trusted-ca-bundle\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.377596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.377593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-config\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.377723 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.377650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-serving-cert\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.377723 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.377668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-oauth-config\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.478870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.478838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-config\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.478880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-serving-cert\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.478903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-oauth-config\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.478930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-oauth-serving-cert\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.478948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ck8g\" (UniqueName: \"kubernetes.io/projected/016d9fb9-4668-4a37-bb01-9871dc33f6ce-kube-api-access-5ck8g\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.478965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-service-ca\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.478983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-trusted-ca-bundle\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479734 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.479670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-oauth-serving-cert\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479734 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.479673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-config\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.479990 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.479902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-service-ca\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.480136 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.480113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-trusted-ca-bundle\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.481506 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.481479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-serving-cert\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.481635 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.481557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-oauth-config\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.488043 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.488024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ck8g\" (UniqueName: \"kubernetes.io/projected/016d9fb9-4668-4a37-bb01-9871dc33f6ce-kube-api-access-5ck8g\") pod \"console-76c8c8844d-t5xsw\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") " pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.606908 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.606816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:37.883362 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:37.883294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c8c8844d-t5xsw"]
Sep 16 12:04:37.886130 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:04:37.886100    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016d9fb9_4668_4a37_bb01_9871dc33f6ce.slice/crio-f8b060ca9435d7b62275fd4b3ae5263cdbbf52750bbcde2f04ef38ee15fadc02 WatchSource:0}: Error finding container f8b060ca9435d7b62275fd4b3ae5263cdbbf52750bbcde2f04ef38ee15fadc02: Status 404 returned error can't find the container with id f8b060ca9435d7b62275fd4b3ae5263cdbbf52750bbcde2f04ef38ee15fadc02
Sep 16 12:04:38.732270 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:38.732235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c8c8844d-t5xsw" event={"ID":"016d9fb9-4668-4a37-bb01-9871dc33f6ce","Type":"ContainerStarted","Data":"1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489"}
Sep 16 12:04:38.732681 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:38.732284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c8c8844d-t5xsw" event={"ID":"016d9fb9-4668-4a37-bb01-9871dc33f6ce","Type":"ContainerStarted","Data":"f8b060ca9435d7b62275fd4b3ae5263cdbbf52750bbcde2f04ef38ee15fadc02"}
Sep 16 12:04:38.750070 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:38.750025    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76c8c8844d-t5xsw" podStartSLOduration=1.7500105110000002 podStartE2EDuration="1.750010511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:38.748658932 +0000 UTC m=+168.385983761" watchObservedRunningTime="2026-09-16 12:04:38.750010511 +0000 UTC m=+168.387335349"
Sep 16 12:04:47.607614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:47.607507    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:47.607614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:47.607591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:47.612061 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:47.612032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:47.773946 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:47.773920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:04:47.822569 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:04:47.822528    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5456c78c44-rn7bl"]
Sep 16 12:05:12.845480 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:12.845437    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5456c78c44-rn7bl" podUID="33897d11-6d61-4156-b659-a431703acf91" containerName="console" containerID="cri-o://46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1" gracePeriod=15
Sep 16 12:05:13.095314 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.095289    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:05:13.256937 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.256901    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-console-config\") pod \"33897d11-6d61-4156-b659-a431703acf91\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") "
Sep 16 12:05:13.256937 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.256937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-service-ca\") pod \"33897d11-6d61-4156-b659-a431703acf91\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") "
Sep 16 12:05:13.257164 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-oauth-config\") pod \"33897d11-6d61-4156-b659-a431703acf91\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") "
Sep 16 12:05:13.257164 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-trusted-ca-bundle\") pod \"33897d11-6d61-4156-b659-a431703acf91\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") "
Sep 16 12:05:13.257269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-oauth-serving-cert\") pod \"33897d11-6d61-4156-b659-a431703acf91\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") "
Sep 16 12:05:13.257269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-serving-cert\") pod \"33897d11-6d61-4156-b659-a431703acf91\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") "
Sep 16 12:05:13.257269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zgd6\" (UniqueName: \"kubernetes.io/projected/33897d11-6d61-4156-b659-a431703acf91-kube-api-access-8zgd6\") pod \"33897d11-6d61-4156-b659-a431703acf91\" (UID: \"33897d11-6d61-4156-b659-a431703acf91\") "
Sep 16 12:05:13.257454 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257430    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-console-config" (OuterVolumeSpecName: "console-config") pod "33897d11-6d61-4156-b659-a431703acf91" (UID: "33897d11-6d61-4156-b659-a431703acf91"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:05:13.257503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-service-ca" (OuterVolumeSpecName: "service-ca") pod "33897d11-6d61-4156-b659-a431703acf91" (UID: "33897d11-6d61-4156-b659-a431703acf91"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:05:13.257571 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257545    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-console-config\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:05:13.257612 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "33897d11-6d61-4156-b659-a431703acf91" (UID: "33897d11-6d61-4156-b659-a431703acf91"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:05:13.257731 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.257715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "33897d11-6d61-4156-b659-a431703acf91" (UID: "33897d11-6d61-4156-b659-a431703acf91"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:05:13.259588 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.259564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "33897d11-6d61-4156-b659-a431703acf91" (UID: "33897d11-6d61-4156-b659-a431703acf91"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:05:13.259694 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.259605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33897d11-6d61-4156-b659-a431703acf91-kube-api-access-8zgd6" (OuterVolumeSpecName: "kube-api-access-8zgd6") pod "33897d11-6d61-4156-b659-a431703acf91" (UID: "33897d11-6d61-4156-b659-a431703acf91"). InnerVolumeSpecName "kube-api-access-8zgd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:05:13.259734 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.259682    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "33897d11-6d61-4156-b659-a431703acf91" (UID: "33897d11-6d61-4156-b659-a431703acf91"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:05:13.358449 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.358417    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zgd6\" (UniqueName: \"kubernetes.io/projected/33897d11-6d61-4156-b659-a431703acf91-kube-api-access-8zgd6\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:05:13.358449 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.358447    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:05:13.358449 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.358457    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-oauth-config\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:05:13.358687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.358467    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-trusted-ca-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:05:13.358687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.358475    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33897d11-6d61-4156-b659-a431703acf91-oauth-serving-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:05:13.358687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.358483    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33897d11-6d61-4156-b659-a431703acf91-console-serving-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:05:13.865073 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.865035    2578 generic.go:356] "Generic (PLEG): container finished" podID="33897d11-6d61-4156-b659-a431703acf91" containerID="46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1" exitCode=0
Sep 16 12:05:13.865615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.865081    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5456c78c44-rn7bl"
Sep 16 12:05:13.865615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.865119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5456c78c44-rn7bl" event={"ID":"33897d11-6d61-4156-b659-a431703acf91","Type":"ContainerDied","Data":"46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1"}
Sep 16 12:05:13.865615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.865209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5456c78c44-rn7bl" event={"ID":"33897d11-6d61-4156-b659-a431703acf91","Type":"ContainerDied","Data":"3f7832c590cac441b7272d303cc653dbedd0d0b7f54a94963b58714fca5ff19e"}
Sep 16 12:05:13.865615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.865231    2578 scope.go:117] "RemoveContainer" containerID="46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1"
Sep 16 12:05:13.873747 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.873730    2578 scope.go:117] "RemoveContainer" containerID="46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1"
Sep 16 12:05:13.874018 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:05:13.874000    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1\": container with ID starting with 46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1 not found: ID does not exist" containerID="46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1"
Sep 16 12:05:13.874065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.874026    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1"} err="failed to get container status \"46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1\": rpc error: code = NotFound desc = could not find container \"46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1\": container with ID starting with 46e572e931d6525685653c889f734a7c49549fdf54d3c076ce512768e48bbfd1 not found: ID does not exist"
Sep 16 12:05:13.886971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.886943    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5456c78c44-rn7bl"]
Sep 16 12:05:13.890982 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:13.890961    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5456c78c44-rn7bl"]
Sep 16 12:05:14.883472 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:14.883439    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33897d11-6d61-4156-b659-a431703acf91" path="/var/lib/kubelet/pods/33897d11-6d61-4156-b659-a431703acf91/volumes"
Sep 16 12:05:46.862271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.862228    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"]
Sep 16 12:05:46.862746 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.862542    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33897d11-6d61-4156-b659-a431703acf91" containerName="console"
Sep 16 12:05:46.862746 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.862555    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33897d11-6d61-4156-b659-a431703acf91" containerName="console"
Sep 16 12:05:46.862746 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.862655    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33897d11-6d61-4156-b659-a431703acf91" containerName="console"
Sep 16 12:05:46.864845 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.864829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:46.867411 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.867386    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:05:46.867568 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.867484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-544wn\""
Sep 16 12:05:46.868477 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.868458    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:05:46.875000 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.874970    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"]
Sep 16 12:05:46.914143 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.914107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:46.914143 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.914150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:46.914369 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:46.914193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjsc\" (UniqueName: \"kubernetes.io/projected/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-kube-api-access-qqjsc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:47.015288 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.015251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjsc\" (UniqueName: \"kubernetes.io/projected/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-kube-api-access-qqjsc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:47.015469 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.015301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:47.015469 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.015416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:47.015753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.015732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:47.015796 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.015757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:47.024057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.024029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjsc\" (UniqueName: \"kubernetes.io/projected/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-kube-api-access-qqjsc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:47.174433 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.174330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:05:47.444265 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.444237    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"]
Sep 16 12:05:47.446386 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:05:47.446355    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3b0cac_dd28_468b_b0c1_7ffbb4f163fb.slice/crio-093388e8efbf4b9da3421aca9238be3387fa76c072a9ab1d264d1ce6ccacc255 WatchSource:0}: Error finding container 093388e8efbf4b9da3421aca9238be3387fa76c072a9ab1d264d1ce6ccacc255: Status 404 returned error can't find the container with id 093388e8efbf4b9da3421aca9238be3387fa76c072a9ab1d264d1ce6ccacc255
Sep 16 12:05:47.994989 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:47.994952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n" event={"ID":"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb","Type":"ContainerStarted","Data":"093388e8efbf4b9da3421aca9238be3387fa76c072a9ab1d264d1ce6ccacc255"}
Sep 16 12:05:55.028019 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:55.027984    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerID="67d9c20eefa0d80fa6741a8e96631cb39fac58bc13133d134e02682ab4f40e34" exitCode=0
Sep 16 12:05:55.028441 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:55.028039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n" event={"ID":"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb","Type":"ContainerDied","Data":"67d9c20eefa0d80fa6741a8e96631cb39fac58bc13133d134e02682ab4f40e34"}
Sep 16 12:05:59.045229 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:59.045191    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerID="834dcbaef7cdc003766417bbd74d78a57c84b9f982133c009206c1d6cddef3aa" exitCode=0
Sep 16 12:05:59.045729 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:05:59.045255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n" event={"ID":"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb","Type":"ContainerDied","Data":"834dcbaef7cdc003766417bbd74d78a57c84b9f982133c009206c1d6cddef3aa"}
Sep 16 12:06:07.081255 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:07.081215    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerID="1cb26559d449491fdb0a15e1d978c83869570cc55b1081790f1a2a90acff1e88" exitCode=0
Sep 16 12:06:07.081713 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:07.081305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n" event={"ID":"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb","Type":"ContainerDied","Data":"1cb26559d449491fdb0a15e1d978c83869570cc55b1081790f1a2a90acff1e88"}
Sep 16 12:06:08.213324 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.213298    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:06:08.287605 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.287564    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-util\") pod \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") "
Sep 16 12:06:08.287803 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.287619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqjsc\" (UniqueName: \"kubernetes.io/projected/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-kube-api-access-qqjsc\") pod \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") "
Sep 16 12:06:08.287803 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.287654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-bundle\") pod \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\" (UID: \"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb\") "
Sep 16 12:06:08.288314 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.288285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-bundle" (OuterVolumeSpecName: "bundle") pod "ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" (UID: "ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:06:08.290009 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.289989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-kube-api-access-qqjsc" (OuterVolumeSpecName: "kube-api-access-qqjsc") pod "ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" (UID: "ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb"). InnerVolumeSpecName "kube-api-access-qqjsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:06:08.291865 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.291838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-util" (OuterVolumeSpecName: "util") pod "ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" (UID: "ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:06:08.388646 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.388543    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:08.388646 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.388583    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-util\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:08.388646 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:08.388593    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjsc\" (UniqueName: \"kubernetes.io/projected/ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb-kube-api-access-qqjsc\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:09.090935 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:09.090908    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n"
Sep 16 12:06:09.091099 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:09.090903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j8h5n" event={"ID":"ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb","Type":"ContainerDied","Data":"093388e8efbf4b9da3421aca9238be3387fa76c072a9ab1d264d1ce6ccacc255"}
Sep 16 12:06:09.091099 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:09.091038    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="093388e8efbf4b9da3421aca9238be3387fa76c072a9ab1d264d1ce6ccacc255"
Sep 16 12:06:28.326926 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.326837    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b8s62"]
Sep 16 12:06:28.327407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.327225    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerName="util"
Sep 16 12:06:28.327407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.327243    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerName="util"
Sep 16 12:06:28.327407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.327270    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerName="extract"
Sep 16 12:06:28.327407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.327279    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerName="extract"
Sep 16 12:06:28.327407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.327294    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerName="pull"
Sep 16 12:06:28.327407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.327302    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerName="pull"
Sep 16 12:06:28.327407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.327369    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3b0cac-dd28-468b-b0c1-7ffbb4f163fb" containerName="extract"
Sep 16 12:06:28.329254 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.329235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:28.332292 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.332273    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 12:06:28.332403 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.332276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vffpz\""
Sep 16 12:06:28.333316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.333301    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 12:06:28.339188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.339153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b8s62"]
Sep 16 12:06:28.456343 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.456301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b8s62\" (UID: \"e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:28.456554 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.456443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnlgn\" (UniqueName: \"kubernetes.io/projected/e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b-kube-api-access-tnlgn\") pod \"cert-manager-webhook-6486ffd997-b8s62\" (UID: \"e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:28.557220 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.557179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnlgn\" (UniqueName: \"kubernetes.io/projected/e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b-kube-api-access-tnlgn\") pod \"cert-manager-webhook-6486ffd997-b8s62\" (UID: \"e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:28.557406 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.557228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b8s62\" (UID: \"e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:28.573632 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.573595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b8s62\" (UID: \"e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:28.575746 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.575722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnlgn\" (UniqueName: \"kubernetes.io/projected/e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b-kube-api-access-tnlgn\") pod \"cert-manager-webhook-6486ffd997-b8s62\" (UID: \"e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:28.650845 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.650740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:28.824009 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.823976    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"]
Sep 16 12:06:28.826426 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.826411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:28.829863 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.829842    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:06:28.829958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.829842    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:06:28.831009 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.830990    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-544wn\""
Sep 16 12:06:28.839199 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.839176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"]
Sep 16 12:06:28.934945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.934772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b8s62"]
Sep 16 12:06:28.937811 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:06:28.937787    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0fc4a5c_0a06_4b31_b086_e89ef0e5d61b.slice/crio-43c6845f89dc78b2b204ae0a5d07c7f4dcb3ba9bcfa0c13096449e41f77d82cc WatchSource:0}: Error finding container 43c6845f89dc78b2b204ae0a5d07c7f4dcb3ba9bcfa0c13096449e41f77d82cc: Status 404 returned error can't find the container with id 43c6845f89dc78b2b204ae0a5d07c7f4dcb3ba9bcfa0c13096449e41f77d82cc
Sep 16 12:06:28.961034 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.961006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:28.961180 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.961067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdf5n\" (UniqueName: \"kubernetes.io/projected/fce657e0-f433-4aa9-b711-e03d01caeca3-kube-api-access-mdf5n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:28.961180 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:28.961155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:29.062120 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.062079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdf5n\" (UniqueName: \"kubernetes.io/projected/fce657e0-f433-4aa9-b711-e03d01caeca3-kube-api-access-mdf5n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:29.062294 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.062133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:29.062294 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.062167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:29.062537 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.062497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:29.062616 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.062593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:29.074466 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.074439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdf5n\" (UniqueName: \"kubernetes.io/projected/fce657e0-f433-4aa9-b711-e03d01caeca3-kube-api-access-mdf5n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:29.136287 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.136261    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:29.169647 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.169607    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62" event={"ID":"e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b","Type":"ContainerStarted","Data":"43c6845f89dc78b2b204ae0a5d07c7f4dcb3ba9bcfa0c13096449e41f77d82cc"}
Sep 16 12:06:29.405430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:29.405391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"]
Sep 16 12:06:29.407918 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:06:29.407881    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce657e0_f433_4aa9_b711_e03d01caeca3.slice/crio-b07583fe694a1c3b843509424cf435ca351f5d6bed464739403a06d798243e4f WatchSource:0}: Error finding container b07583fe694a1c3b843509424cf435ca351f5d6bed464739403a06d798243e4f: Status 404 returned error can't find the container with id b07583fe694a1c3b843509424cf435ca351f5d6bed464739403a06d798243e4f
Sep 16 12:06:30.177330 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:30.177291    2578 generic.go:356] "Generic (PLEG): container finished" podID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerID="9cf49f3a86e90b26ab4536f89ac22d75567dca06578c56d0a3994a7bf08f1a51" exitCode=0
Sep 16 12:06:30.177543 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:30.177395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9" event={"ID":"fce657e0-f433-4aa9-b711-e03d01caeca3","Type":"ContainerDied","Data":"9cf49f3a86e90b26ab4536f89ac22d75567dca06578c56d0a3994a7bf08f1a51"}
Sep 16 12:06:30.177543 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:30.177455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9" event={"ID":"fce657e0-f433-4aa9-b711-e03d01caeca3","Type":"ContainerStarted","Data":"b07583fe694a1c3b843509424cf435ca351f5d6bed464739403a06d798243e4f"}
Sep 16 12:06:33.192625 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:33.192583    2578 generic.go:356] "Generic (PLEG): container finished" podID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerID="4d7e8722124d713f89e599de8624a7a71cc8c819d7b34210976de3c086e2cac4" exitCode=0
Sep 16 12:06:33.193112 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:33.192671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9" event={"ID":"fce657e0-f433-4aa9-b711-e03d01caeca3","Type":"ContainerDied","Data":"4d7e8722124d713f89e599de8624a7a71cc8c819d7b34210976de3c086e2cac4"}
Sep 16 12:06:33.194376 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:33.194297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62" event={"ID":"e0fc4a5c-0a06-4b31-b086-e89ef0e5d61b","Type":"ContainerStarted","Data":"85bbded6bde238bf69a966ba92b7c520938e3499c73291ce15f88e7b8dc5f184"}
Sep 16 12:06:33.194451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:33.194420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:33.230308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:33.230257    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62" podStartSLOduration=1.665454612 podStartE2EDuration="5.230239184s" totalImagesPullingTime="3.564784572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:06:33.228475557 +0000 UTC m=+282.865800400" watchObservedRunningTime="2026-09-16 12:06:33.230239184 +0000 UTC m=+282.867564022"
Sep 16 12:06:34.200831 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:34.200796    2578 generic.go:356] "Generic (PLEG): container finished" podID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerID="197b84fd79c623d450c319db38136974b7c9ac07e2c5cf5eff6c66cf51cbac12" exitCode=0
Sep 16 12:06:34.201238 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:34.200877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9" event={"ID":"fce657e0-f433-4aa9-b711-e03d01caeca3","Type":"ContainerDied","Data":"197b84fd79c623d450c319db38136974b7c9ac07e2c5cf5eff6c66cf51cbac12"}
Sep 16 12:06:35.325759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.325732    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:35.416354 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.416317    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdf5n\" (UniqueName: \"kubernetes.io/projected/fce657e0-f433-4aa9-b711-e03d01caeca3-kube-api-access-mdf5n\") pod \"fce657e0-f433-4aa9-b711-e03d01caeca3\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") "
Sep 16 12:06:35.416354 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.416356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-util\") pod \"fce657e0-f433-4aa9-b711-e03d01caeca3\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") "
Sep 16 12:06:35.416631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.416394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-bundle\") pod \"fce657e0-f433-4aa9-b711-e03d01caeca3\" (UID: \"fce657e0-f433-4aa9-b711-e03d01caeca3\") "
Sep 16 12:06:35.416843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.416820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-bundle" (OuterVolumeSpecName: "bundle") pod "fce657e0-f433-4aa9-b711-e03d01caeca3" (UID: "fce657e0-f433-4aa9-b711-e03d01caeca3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:06:35.418603 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.418578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce657e0-f433-4aa9-b711-e03d01caeca3-kube-api-access-mdf5n" (OuterVolumeSpecName: "kube-api-access-mdf5n") pod "fce657e0-f433-4aa9-b711-e03d01caeca3" (UID: "fce657e0-f433-4aa9-b711-e03d01caeca3"). InnerVolumeSpecName "kube-api-access-mdf5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:06:35.421564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.421542    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-util" (OuterVolumeSpecName: "util") pod "fce657e0-f433-4aa9-b711-e03d01caeca3" (UID: "fce657e0-f433-4aa9-b711-e03d01caeca3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:06:35.517277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.517188    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdf5n\" (UniqueName: \"kubernetes.io/projected/fce657e0-f433-4aa9-b711-e03d01caeca3-kube-api-access-mdf5n\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:35.517277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.517222    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-util\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:35.517277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:35.517231    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fce657e0-f433-4aa9-b711-e03d01caeca3-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:36.210638 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:36.210594    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9"
Sep 16 12:06:36.210638 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:36.210614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhbnr9" event={"ID":"fce657e0-f433-4aa9-b711-e03d01caeca3","Type":"ContainerDied","Data":"b07583fe694a1c3b843509424cf435ca351f5d6bed464739403a06d798243e4f"}
Sep 16 12:06:36.210852 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:36.210664    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b07583fe694a1c3b843509424cf435ca351f5d6bed464739403a06d798243e4f"
Sep 16 12:06:39.203030 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:39.203002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-b8s62"
Sep 16 12:06:46.763120 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.763087    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"]
Sep 16 12:06:46.763551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.763381    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerName="util"
Sep 16 12:06:46.763551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.763391    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerName="util"
Sep 16 12:06:46.763551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.763409    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerName="extract"
Sep 16 12:06:46.763551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.763414    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerName="extract"
Sep 16 12:06:46.763551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.763425    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerName="pull"
Sep 16 12:06:46.763551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.763430    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerName="pull"
Sep 16 12:06:46.763551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.763480    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce657e0-f433-4aa9-b711-e03d01caeca3" containerName="extract"
Sep 16 12:06:46.765581 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.765564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:46.768544 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.768506    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:06:46.768649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.768505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:06:46.769722 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.769708    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-544wn\""
Sep 16 12:06:46.775809 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.775789    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"]
Sep 16 12:06:46.913346 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.913309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:46.913546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.913367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw5gm\" (UniqueName: \"kubernetes.io/projected/6cd06553-63a1-4af3-835f-7917c1556b40-kube-api-access-qw5gm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:46.913546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:46.913385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:47.014752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:47.014685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:47.014752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:47.014746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw5gm\" (UniqueName: \"kubernetes.io/projected/6cd06553-63a1-4af3-835f-7917c1556b40-kube-api-access-qw5gm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:47.014892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:47.014763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:47.015040 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:47.015022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:47.015116 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:47.015102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:47.024146 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:47.024124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw5gm\" (UniqueName: \"kubernetes.io/projected/6cd06553-63a1-4af3-835f-7917c1556b40-kube-api-access-qw5gm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:47.075305 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:47.075269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:47.345598 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:47.345573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"]
Sep 16 12:06:47.348074 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:06:47.348052    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd06553_63a1_4af3_835f_7917c1556b40.slice/crio-57ca7c3262964df03f00a381a63371fa341402f0194dc01ff1f6451a1d4b785a WatchSource:0}: Error finding container 57ca7c3262964df03f00a381a63371fa341402f0194dc01ff1f6451a1d4b785a: Status 404 returned error can't find the container with id 57ca7c3262964df03f00a381a63371fa341402f0194dc01ff1f6451a1d4b785a
Sep 16 12:06:48.260180 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:48.260148    2578 generic.go:356] "Generic (PLEG): container finished" podID="6cd06553-63a1-4af3-835f-7917c1556b40" containerID="ad3798e5ae593671cc385128bbf726d5b025b2441f24a10c2afb5fa9a1998d6b" exitCode=0
Sep 16 12:06:48.260657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:48.260242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p" event={"ID":"6cd06553-63a1-4af3-835f-7917c1556b40","Type":"ContainerDied","Data":"ad3798e5ae593671cc385128bbf726d5b025b2441f24a10c2afb5fa9a1998d6b"}
Sep 16 12:06:48.260657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:48.260312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p" event={"ID":"6cd06553-63a1-4af3-835f-7917c1556b40","Type":"ContainerStarted","Data":"57ca7c3262964df03f00a381a63371fa341402f0194dc01ff1f6451a1d4b785a"}
Sep 16 12:06:49.265930 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:49.265837    2578 generic.go:356] "Generic (PLEG): container finished" podID="6cd06553-63a1-4af3-835f-7917c1556b40" containerID="8bccc355e5cdc89b888c51a3e6af3e9114562c83d3a7ead479f5a39d490d2167" exitCode=0
Sep 16 12:06:49.266305 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:49.265929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p" event={"ID":"6cd06553-63a1-4af3-835f-7917c1556b40","Type":"ContainerDied","Data":"8bccc355e5cdc89b888c51a3e6af3e9114562c83d3a7ead479f5a39d490d2167"}
Sep 16 12:06:50.272434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:50.272402    2578 generic.go:356] "Generic (PLEG): container finished" podID="6cd06553-63a1-4af3-835f-7917c1556b40" containerID="6b2706cc0341afe39ff04b7e73d4ed633a81c38d833ad899f4a3349b4422d468" exitCode=0
Sep 16 12:06:50.272818 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:50.272470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p" event={"ID":"6cd06553-63a1-4af3-835f-7917c1556b40","Type":"ContainerDied","Data":"6b2706cc0341afe39ff04b7e73d4ed633a81c38d833ad899f4a3349b4422d468"}
Sep 16 12:06:50.775379 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:50.775349    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:06:50.780123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:50.780099    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:06:50.788636 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:50.788610    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:06:51.399229 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.399203    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:51.451042 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.451012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-util\") pod \"6cd06553-63a1-4af3-835f-7917c1556b40\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") "
Sep 16 12:06:51.451212 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.451049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw5gm\" (UniqueName: \"kubernetes.io/projected/6cd06553-63a1-4af3-835f-7917c1556b40-kube-api-access-qw5gm\") pod \"6cd06553-63a1-4af3-835f-7917c1556b40\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") "
Sep 16 12:06:51.451212 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.451073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-bundle\") pod \"6cd06553-63a1-4af3-835f-7917c1556b40\" (UID: \"6cd06553-63a1-4af3-835f-7917c1556b40\") "
Sep 16 12:06:51.451854 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.451830    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-bundle" (OuterVolumeSpecName: "bundle") pod "6cd06553-63a1-4af3-835f-7917c1556b40" (UID: "6cd06553-63a1-4af3-835f-7917c1556b40"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:06:51.453188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.453165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd06553-63a1-4af3-835f-7917c1556b40-kube-api-access-qw5gm" (OuterVolumeSpecName: "kube-api-access-qw5gm") pod "6cd06553-63a1-4af3-835f-7917c1556b40" (UID: "6cd06553-63a1-4af3-835f-7917c1556b40"). InnerVolumeSpecName "kube-api-access-qw5gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:06:51.456830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.456803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-util" (OuterVolumeSpecName: "util") pod "6cd06553-63a1-4af3-835f-7917c1556b40" (UID: "6cd06553-63a1-4af3-835f-7917c1556b40"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:06:51.551668 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.551591    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:51.551668 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.551623    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cd06553-63a1-4af3-835f-7917c1556b40-util\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:51.551668 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:51.551638    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw5gm\" (UniqueName: \"kubernetes.io/projected/6cd06553-63a1-4af3-835f-7917c1556b40-kube-api-access-qw5gm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:06:52.283110 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:52.283081    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p"
Sep 16 12:06:52.283110 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:52.283105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsx77p" event={"ID":"6cd06553-63a1-4af3-835f-7917c1556b40","Type":"ContainerDied","Data":"57ca7c3262964df03f00a381a63371fa341402f0194dc01ff1f6451a1d4b785a"}
Sep 16 12:06:52.283320 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:06:52.283144    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ca7c3262964df03f00a381a63371fa341402f0194dc01ff1f6451a1d4b785a"
Sep 16 12:07:03.993483 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:03.993441    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"]
Sep 16 12:07:03.993906 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:03.993816    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd06553-63a1-4af3-835f-7917c1556b40" containerName="util"
Sep 16 12:07:03.993906 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:03.993831    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd06553-63a1-4af3-835f-7917c1556b40" containerName="util"
Sep 16 12:07:03.993906 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:03.993839    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd06553-63a1-4af3-835f-7917c1556b40" containerName="pull"
Sep 16 12:07:03.993906 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:03.993846    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd06553-63a1-4af3-835f-7917c1556b40" containerName="pull"
Sep 16 12:07:03.993906 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:03.993867    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd06553-63a1-4af3-835f-7917c1556b40" containerName="extract"
Sep 16 12:07:03.993906 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:03.993872    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd06553-63a1-4af3-835f-7917c1556b40" containerName="extract"
Sep 16 12:07:03.994109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:03.993917    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd06553-63a1-4af3-835f-7917c1556b40" containerName="extract"
Sep 16 12:07:04.040025 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.039994    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"]
Sep 16 12:07:04.040184 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.040118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.043294 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.043269    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:07:04.043426 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.043359    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:07:04.043628 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.043609    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-544wn\""
Sep 16 12:07:04.065383 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.065357    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"]
Sep 16 12:07:04.065499 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.065391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"]
Sep 16 12:07:04.065499 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.065484    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.069561 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.069539    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 12:07:04.069782 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.069768    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 12:07:04.069985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.069971    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-22d2l\""
Sep 16 12:07:04.069985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.069978    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 12:07:04.070113 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.070037    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 12:07:04.144197 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.144158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lq8x\" (UniqueName: \"kubernetes.io/projected/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-kube-api-access-4lq8x\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.144370 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.144215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.144370 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.144277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-tmp\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.144370 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.144304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgpz\" (UniqueName: \"kubernetes.io/projected/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-kube-api-access-sxgpz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.144370 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.144326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.144370 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.144352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.144566 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.144418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-manifests\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.144566 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.144459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.244859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.244776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-tmp\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.244859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.244814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgpz\" (UniqueName: \"kubernetes.io/projected/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-kube-api-access-sxgpz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.244859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.244839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.245130 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.244878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.245130 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.244906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-manifests\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.245130 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.244932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.245130 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.245020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lq8x\" (UniqueName: \"kubernetes.io/projected/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-kube-api-access-4lq8x\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.245336 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.245141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.245336 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.245285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.245336 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.245308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-manifests\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.245560 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.245539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.247233 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.247213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-tmp\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.247500 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.247481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.247593 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.247560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.263984 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.263960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lq8x\" (UniqueName: \"kubernetes.io/projected/a5ffc7b4-3b52-402a-a813-d8f8c737f09d-kube-api-access-4lq8x\") pod \"opendatahub-operator-controller-manager-6b54c497cf-pdcp9\" (UID: \"a5ffc7b4-3b52-402a-a813-d8f8c737f09d\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.269828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.269805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgpz\" (UniqueName: \"kubernetes.io/projected/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-kube-api-access-sxgpz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.350784 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.350754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:04.375497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.375480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:04.644604 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.644572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"]
Sep 16 12:07:04.646785 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:07:04.646758    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e37881b_9ee2_4642_9aa2_9a4b82c8696d.slice/crio-f3b105da81ad5e4cc55f98ff4a66740ea2999f8a3f7d877a7ecd5c4efe620577 WatchSource:0}: Error finding container f3b105da81ad5e4cc55f98ff4a66740ea2999f8a3f7d877a7ecd5c4efe620577: Status 404 returned error can't find the container with id f3b105da81ad5e4cc55f98ff4a66740ea2999f8a3f7d877a7ecd5c4efe620577
Sep 16 12:07:04.679704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.679678    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"]
Sep 16 12:07:04.681248 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:07:04.681222    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ffc7b4_3b52_402a_a813_d8f8c737f09d.slice/crio-ff425b93ff7998b4c80bc3d0c73914e3b1c24861ff370baa870be451c1d952f6 WatchSource:0}: Error finding container ff425b93ff7998b4c80bc3d0c73914e3b1c24861ff370baa870be451c1d952f6: Status 404 returned error can't find the container with id ff425b93ff7998b4c80bc3d0c73914e3b1c24861ff370baa870be451c1d952f6
Sep 16 12:07:04.683708 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:04.683691    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:07:05.339034 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:05.338993    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerID="35516e5c144dd819d2328badaed2913a05e7e2318a6f9bdfb0d16f11de6c034e" exitCode=0
Sep 16 12:07:05.339644 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:05.339091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw" event={"ID":"2e37881b-9ee2-4642-9aa2-9a4b82c8696d","Type":"ContainerDied","Data":"35516e5c144dd819d2328badaed2913a05e7e2318a6f9bdfb0d16f11de6c034e"}
Sep 16 12:07:05.339644 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:05.339138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw" event={"ID":"2e37881b-9ee2-4642-9aa2-9a4b82c8696d","Type":"ContainerStarted","Data":"f3b105da81ad5e4cc55f98ff4a66740ea2999f8a3f7d877a7ecd5c4efe620577"}
Sep 16 12:07:05.343123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:05.343086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9" event={"ID":"a5ffc7b4-3b52-402a-a813-d8f8c737f09d","Type":"ContainerStarted","Data":"ff425b93ff7998b4c80bc3d0c73914e3b1c24861ff370baa870be451c1d952f6"}
Sep 16 12:07:08.362014 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:08.361978    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerID="eaeca411be6c3746750af44a317838b319e10f79f579a85aaac4cd4142c2ab43" exitCode=0
Sep 16 12:07:08.362480 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:08.362062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw" event={"ID":"2e37881b-9ee2-4642-9aa2-9a4b82c8696d","Type":"ContainerDied","Data":"eaeca411be6c3746750af44a317838b319e10f79f579a85aaac4cd4142c2ab43"}
Sep 16 12:07:08.363674 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:08.363649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9" event={"ID":"a5ffc7b4-3b52-402a-a813-d8f8c737f09d","Type":"ContainerStarted","Data":"b95b7da9be01f6f31eda6ba7bc58cc75f72558ab0c8d0692167c8369f258d50b"}
Sep 16 12:07:09.370791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:09.370753    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerID="8e62ed145e4a3024ea3f5397d894ad971654d9f9a3bdb25b2619edc29146d21c" exitCode=0
Sep 16 12:07:09.371222 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:09.370839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw" event={"ID":"2e37881b-9ee2-4642-9aa2-9a4b82c8696d","Type":"ContainerDied","Data":"8e62ed145e4a3024ea3f5397d894ad971654d9f9a3bdb25b2619edc29146d21c"}
Sep 16 12:07:09.372451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:09.372425    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5ffc7b4-3b52-402a-a813-d8f8c737f09d" containerID="b95b7da9be01f6f31eda6ba7bc58cc75f72558ab0c8d0692167c8369f258d50b" exitCode=0
Sep 16 12:07:09.372564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:09.372532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9" event={"ID":"a5ffc7b4-3b52-402a-a813-d8f8c737f09d","Type":"ContainerDied","Data":"b95b7da9be01f6f31eda6ba7bc58cc75f72558ab0c8d0692167c8369f258d50b"}
Sep 16 12:07:10.378969 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.378929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9" event={"ID":"a5ffc7b4-3b52-402a-a813-d8f8c737f09d","Type":"ContainerStarted","Data":"614c076513b4a2df4afd09f7ad2b1deabd8239c836047e31b60a9128a57c15bd"}
Sep 16 12:07:10.379451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.379277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:10.402755 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.402707    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9" podStartSLOduration=2.985381013 podStartE2EDuration="7.402688502s" totalImagesPullingTime="3.325397029s" totalInitContainerRuntime="1.09191046s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:10.400820811 +0000 UTC m=+320.038145651" watchObservedRunningTime="2026-09-16 12:07:10.402688502 +0000 UTC m=+320.040013340"
Sep 16 12:07:10.509827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.509798    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:10.598742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.598705    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-util\") pod \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") "
Sep 16 12:07:10.598742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.598747    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-bundle\") pod \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") "
Sep 16 12:07:10.598997 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.598784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxgpz\" (UniqueName: \"kubernetes.io/projected/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-kube-api-access-sxgpz\") pod \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\" (UID: \"2e37881b-9ee2-4642-9aa2-9a4b82c8696d\") "
Sep 16 12:07:10.599564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.599540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-bundle" (OuterVolumeSpecName: "bundle") pod "2e37881b-9ee2-4642-9aa2-9a4b82c8696d" (UID: "2e37881b-9ee2-4642-9aa2-9a4b82c8696d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:07:10.601022 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.601002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-kube-api-access-sxgpz" (OuterVolumeSpecName: "kube-api-access-sxgpz") pod "2e37881b-9ee2-4642-9aa2-9a4b82c8696d" (UID: "2e37881b-9ee2-4642-9aa2-9a4b82c8696d"). InnerVolumeSpecName "kube-api-access-sxgpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:07:10.603600 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.603559    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-util" (OuterVolumeSpecName: "util") pod "2e37881b-9ee2-4642-9aa2-9a4b82c8696d" (UID: "2e37881b-9ee2-4642-9aa2-9a4b82c8696d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:07:10.699806 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.699710    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-util\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:10.699806 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.699744    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:10.699806 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:10.699759    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgpz\" (UniqueName: \"kubernetes.io/projected/2e37881b-9ee2-4642-9aa2-9a4b82c8696d-kube-api-access-sxgpz\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:11.384839 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:11.384810    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw"
Sep 16 12:07:11.384839 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:11.384830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b06dftw" event={"ID":"2e37881b-9ee2-4642-9aa2-9a4b82c8696d","Type":"ContainerDied","Data":"f3b105da81ad5e4cc55f98ff4a66740ea2999f8a3f7d877a7ecd5c4efe620577"}
Sep 16 12:07:11.385346 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:11.384881    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b105da81ad5e4cc55f98ff4a66740ea2999f8a3f7d877a7ecd5c4efe620577"
Sep 16 12:07:21.387464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:21.387431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-pdcp9"
Sep 16 12:07:24.061719 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.061684    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"]
Sep 16 12:07:24.062114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.061981    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerName="util"
Sep 16 12:07:24.062114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.061991    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerName="util"
Sep 16 12:07:24.062114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.062001    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerName="extract"
Sep 16 12:07:24.062114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.062006    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerName="extract"
Sep 16 12:07:24.062114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.062013    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerName="pull"
Sep 16 12:07:24.062114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.062018    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerName="pull"
Sep 16 12:07:24.062114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.062065    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e37881b-9ee2-4642-9aa2-9a4b82c8696d" containerName="extract"
Sep 16 12:07:24.068697 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.068678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.074418 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.074392    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 12:07:24.074573 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.074446    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 12:07:24.095299 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.095261    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"]
Sep 16 12:07:24.200541 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.200496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57927c1d-b69f-42e5-943f-41a163bdd5a2-tls-certs\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.200691 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.200549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgh77\" (UniqueName: \"kubernetes.io/projected/57927c1d-b69f-42e5-943f-41a163bdd5a2-kube-api-access-hgh77\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.200691 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.200623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57927c1d-b69f-42e5-943f-41a163bdd5a2-tmp\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.301406 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.301365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh77\" (UniqueName: \"kubernetes.io/projected/57927c1d-b69f-42e5-943f-41a163bdd5a2-kube-api-access-hgh77\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.301631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.301425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57927c1d-b69f-42e5-943f-41a163bdd5a2-tmp\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.301631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.301535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57927c1d-b69f-42e5-943f-41a163bdd5a2-tls-certs\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.303959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.303931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57927c1d-b69f-42e5-943f-41a163bdd5a2-tmp\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.304062 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.304047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57927c1d-b69f-42e5-943f-41a163bdd5a2-tls-certs\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.320113 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.320055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh77\" (UniqueName: \"kubernetes.io/projected/57927c1d-b69f-42e5-943f-41a163bdd5a2-kube-api-access-hgh77\") pod \"kube-auth-proxy-d74dbbc-hbzgj\" (UID: \"57927c1d-b69f-42e5-943f-41a163bdd5a2\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.378091 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.378064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"
Sep 16 12:07:24.659869 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:24.659824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj"]
Sep 16 12:07:24.662316 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:07:24.662283    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57927c1d_b69f_42e5_943f_41a163bdd5a2.slice/crio-ec91a9fc112e05c680b78980488143cd7a211c1a8b414cae7b58ad9497517653 WatchSource:0}: Error finding container ec91a9fc112e05c680b78980488143cd7a211c1a8b414cae7b58ad9497517653: Status 404 returned error can't find the container with id ec91a9fc112e05c680b78980488143cd7a211c1a8b414cae7b58ad9497517653
Sep 16 12:07:25.453228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.453178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj" event={"ID":"57927c1d-b69f-42e5-943f-41a163bdd5a2","Type":"ContainerStarted","Data":"ec91a9fc112e05c680b78980488143cd7a211c1a8b414cae7b58ad9497517653"}
Sep 16 12:07:25.938863 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.938830    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"]
Sep 16 12:07:25.942596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.942572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:25.945721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.945696    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5z849\""
Sep 16 12:07:25.945848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.945696    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:07:25.946273 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.946221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 12:07:25.946273 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.946245    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 12:07:25.947373 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.947174    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 12:07:25.952691 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:25.952576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"]
Sep 16 12:07:26.016367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.016330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.016584 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.016378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvft8\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-kube-api-access-tvft8\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.016584 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.016415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.016584 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.016434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.016584 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.016467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.016584 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.016549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.016891 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.016597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.117344 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.117312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.117344 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.117349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvft8\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-kube-api-access-tvft8\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.117622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.117395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.117622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.117419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.117622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.117454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.117622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.117530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.117622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.117583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.118385 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.118336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.120377 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.120329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.120716 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.120666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.120716 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.120693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.121002 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.120977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.127155 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.127129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.127548 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.127503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvft8\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-kube-api-access-tvft8\") pod \"istiod-openshift-gateway-7b7995d5cd-cbh76\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:26.256369 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:26.256336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:27.730718 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:27.730688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"]
Sep 16 12:07:27.732311 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:07:27.732287    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d43f24_182d_42c7_90ec_fa68f6a9ca19.slice/crio-50f1f1aa10ee6c8b34801c9345f016184dfd2a70f2736ef7e8155e7b2d09aeff WatchSource:0}: Error finding container 50f1f1aa10ee6c8b34801c9345f016184dfd2a70f2736ef7e8155e7b2d09aeff: Status 404 returned error can't find the container with id 50f1f1aa10ee6c8b34801c9345f016184dfd2a70f2736ef7e8155e7b2d09aeff
Sep 16 12:07:28.469350 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:28.469298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76" event={"ID":"c0d43f24-182d-42c7-90ec-fa68f6a9ca19","Type":"ContainerStarted","Data":"50f1f1aa10ee6c8b34801c9345f016184dfd2a70f2736ef7e8155e7b2d09aeff"}
Sep 16 12:07:28.471414 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:28.471385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj" event={"ID":"57927c1d-b69f-42e5-943f-41a163bdd5a2","Type":"ContainerStarted","Data":"cf394b21ee62e7fbbb8419f75ed9feeb0525acd0f2ea4d05fca226a6ca6a1609"}
Sep 16 12:07:28.489131 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:28.489088    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hbzgj" podStartSLOduration=1.6325368070000001 podStartE2EDuration="4.489074394s" totalImagesPullingTime="2.856537587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:28.48793114 +0000 UTC m=+338.125255979" watchObservedRunningTime="2026-09-16 12:07:28.489074394 +0000 UTC m=+338.126399262"
Sep 16 12:07:30.176591 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:30.176552    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:07:30.176879 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:30.176621    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:07:30.482810 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:30.482776    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76" event={"ID":"c0d43f24-182d-42c7-90ec-fa68f6a9ca19","Type":"ContainerStarted","Data":"5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da"}
Sep 16 12:07:30.483009 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:30.482939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:30.505587 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:30.505545    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76" podStartSLOduration=3.063713396 podStartE2EDuration="5.505531407s" totalImagesPullingTime="2.441818011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:30.503912988 +0000 UTC m=+340.141237823" watchObservedRunningTime="2026-09-16 12:07:30.505531407 +0000 UTC m=+340.142856243"
Sep 16 12:07:31.488850 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:31.488820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:42.540262 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.540184    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"]
Sep 16 12:07:42.543478 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.543463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.546631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.546609    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 12:07:42.546756 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.546609    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lwbjg\""
Sep 16 12:07:42.547326 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.547303    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 12:07:42.547454 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.547384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 12:07:42.547572 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.547485    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 12:07:42.547572 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.547564    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:07:42.552705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.552683    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"]
Sep 16 12:07:42.662598 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.662557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rtz\" (UniqueName: \"kubernetes.io/projected/82b4f555-e233-4ada-92d7-b3e6fa5119d0-kube-api-access-68rtz\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.662772 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.662628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82b4f555-e233-4ada-92d7-b3e6fa5119d0-metrics-cert\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.662772 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.662660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82b4f555-e233-4ada-92d7-b3e6fa5119d0-manager-config\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.662772 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.662678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82b4f555-e233-4ada-92d7-b3e6fa5119d0-cert\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.763994 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.763958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82b4f555-e233-4ada-92d7-b3e6fa5119d0-metrics-cert\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.764178 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.764006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82b4f555-e233-4ada-92d7-b3e6fa5119d0-manager-config\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.764178 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.764028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82b4f555-e233-4ada-92d7-b3e6fa5119d0-cert\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.764178 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.764062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rtz\" (UniqueName: \"kubernetes.io/projected/82b4f555-e233-4ada-92d7-b3e6fa5119d0-kube-api-access-68rtz\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.764861 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.764835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82b4f555-e233-4ada-92d7-b3e6fa5119d0-manager-config\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.766608 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.766585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82b4f555-e233-4ada-92d7-b3e6fa5119d0-cert\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.766710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.766653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82b4f555-e233-4ada-92d7-b3e6fa5119d0-metrics-cert\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.780583 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.780563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rtz\" (UniqueName: \"kubernetes.io/projected/82b4f555-e233-4ada-92d7-b3e6fa5119d0-kube-api-access-68rtz\") pod \"lws-controller-manager-596fbdbcd6-4pxnd\" (UID: \"82b4f555-e233-4ada-92d7-b3e6fa5119d0\") " pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:42.853670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:42.853581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:43.153389 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:43.153365    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"]
Sep 16 12:07:43.154984 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:07:43.154962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b4f555_e233_4ada_92d7_b3e6fa5119d0.slice/crio-5055df30d32f40000ad8b49cd03f7d3a38759b69f5475a1bd32e0e77b02598f7 WatchSource:0}: Error finding container 5055df30d32f40000ad8b49cd03f7d3a38759b69f5475a1bd32e0e77b02598f7: Status 404 returned error can't find the container with id 5055df30d32f40000ad8b49cd03f7d3a38759b69f5475a1bd32e0e77b02598f7
Sep 16 12:07:43.544668 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:43.544636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd" event={"ID":"82b4f555-e233-4ada-92d7-b3e6fa5119d0","Type":"ContainerStarted","Data":"5055df30d32f40000ad8b49cd03f7d3a38759b69f5475a1bd32e0e77b02598f7"}
Sep 16 12:07:45.564228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:45.559068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd" event={"ID":"82b4f555-e233-4ada-92d7-b3e6fa5119d0","Type":"ContainerStarted","Data":"4b3f7e825d0812565be13adcf04539a5f18366b19f3140e7874618a5fc5d3f68"}
Sep 16 12:07:45.564228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:45.560253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:07:45.585467 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:45.585419    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd" podStartSLOduration=1.789321291 podStartE2EDuration="3.585402949s" totalImagesPullingTime="1.796081658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:45.583908073 +0000 UTC m=+355.221232924" watchObservedRunningTime="2026-09-16 12:07:45.585402949 +0000 UTC m=+355.222727787"
Sep 16 12:07:48.547885 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.547855    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"]
Sep 16 12:07:48.548267 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.548060    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76" podUID="c0d43f24-182d-42c7-90ec-fa68f6a9ca19" containerName="discovery" containerID="cri-o://5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da" gracePeriod=30
Sep 16 12:07:48.793644 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.793620    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:48.916786 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.916717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-local-certs\") pod \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") "
Sep 16 12:07:48.916786 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.916763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-kubeconfig\") pod \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") "
Sep 16 12:07:48.917055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.916794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-dns-cert\") pod \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") "
Sep 16 12:07:48.917055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.916846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvft8\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-kube-api-access-tvft8\") pod \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") "
Sep 16 12:07:48.917055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.916870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-cacerts\") pod \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") "
Sep 16 12:07:48.917055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.916949    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-ca-configmap\") pod \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") "
Sep 16 12:07:48.917055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.917003    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-token\") pod \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\" (UID: \"c0d43f24-182d-42c7-90ec-fa68f6a9ca19\") "
Sep 16 12:07:48.917317 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.917294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c0d43f24-182d-42c7-90ec-fa68f6a9ca19" (UID: "c0d43f24-182d-42c7-90ec-fa68f6a9ca19"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:07:48.919358 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.919330    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-cacerts" (OuterVolumeSpecName: "cacerts") pod "c0d43f24-182d-42c7-90ec-fa68f6a9ca19" (UID: "c0d43f24-182d-42c7-90ec-fa68f6a9ca19"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:07:48.919478 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.919371    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c0d43f24-182d-42c7-90ec-fa68f6a9ca19" (UID: "c0d43f24-182d-42c7-90ec-fa68f6a9ca19"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:07:48.919675 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.919644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-local-certs" (OuterVolumeSpecName: "local-certs") pod "c0d43f24-182d-42c7-90ec-fa68f6a9ca19" (UID: "c0d43f24-182d-42c7-90ec-fa68f6a9ca19"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:07:48.919760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.919741    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c0d43f24-182d-42c7-90ec-fa68f6a9ca19" (UID: "c0d43f24-182d-42c7-90ec-fa68f6a9ca19"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:07:48.919916 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.919896    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-kube-api-access-tvft8" (OuterVolumeSpecName: "kube-api-access-tvft8") pod "c0d43f24-182d-42c7-90ec-fa68f6a9ca19" (UID: "c0d43f24-182d-42c7-90ec-fa68f6a9ca19"). InnerVolumeSpecName "kube-api-access-tvft8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:07:48.919970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:48.919909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0d43f24-182d-42c7-90ec-fa68f6a9ca19" (UID: "c0d43f24-182d-42c7-90ec-fa68f6a9ca19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:07:49.018170 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.018136    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-local-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:49.018170 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.018160    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-kubeconfig\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:49.018170 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.018171    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-dns-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:49.018380 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.018184    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvft8\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-kube-api-access-tvft8\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:49.018380 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.018196    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-cacerts\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:49.018380 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.018210    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-csr-ca-configmap\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:49.018380 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.018221    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d43f24-182d-42c7-90ec-fa68f6a9ca19-istio-token\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:07:49.577887 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.577848    2578 generic.go:356] "Generic (PLEG): container finished" podID="c0d43f24-182d-42c7-90ec-fa68f6a9ca19" containerID="5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da" exitCode=0
Sep 16 12:07:49.578300 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.577902    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"
Sep 16 12:07:49.578300 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.577934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76" event={"ID":"c0d43f24-182d-42c7-90ec-fa68f6a9ca19","Type":"ContainerDied","Data":"5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da"}
Sep 16 12:07:49.578300 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.577988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76" event={"ID":"c0d43f24-182d-42c7-90ec-fa68f6a9ca19","Type":"ContainerDied","Data":"50f1f1aa10ee6c8b34801c9345f016184dfd2a70f2736ef7e8155e7b2d09aeff"}
Sep 16 12:07:49.578300 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.578010    2578 scope.go:117] "RemoveContainer" containerID="5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da"
Sep 16 12:07:49.587712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.587686    2578 scope.go:117] "RemoveContainer" containerID="5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da"
Sep 16 12:07:49.587965 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:07:49.587943    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da\": container with ID starting with 5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da not found: ID does not exist" containerID="5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da"
Sep 16 12:07:49.588029 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.587973    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da"} err="failed to get container status \"5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da\": rpc error: code = NotFound desc = could not find container \"5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da\": container with ID starting with 5834bc2bce9065d372c2112f83fd7e37a7e802c46f424a4728896a4fff2141da not found: ID does not exist"
Sep 16 12:07:49.604726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.604680    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"]
Sep 16 12:07:49.608869 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:49.608849    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cbh76"]
Sep 16 12:07:50.884322 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:50.884290    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d43f24-182d-42c7-90ec-fa68f6a9ca19" path="/var/lib/kubelet/pods/c0d43f24-182d-42c7-90ec-fa68f6a9ca19/volumes"
Sep 16 12:07:57.570006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:07:57.569975    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-596fbdbcd6-4pxnd"
Sep 16 12:08:32.238778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:32.238739    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76c8c8844d-t5xsw"]
Sep 16 12:08:39.395708 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.395672    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"]
Sep 16 12:08:39.396182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.396107    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d43f24-182d-42c7-90ec-fa68f6a9ca19" containerName="discovery"
Sep 16 12:08:39.396182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.396126    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d43f24-182d-42c7-90ec-fa68f6a9ca19" containerName="discovery"
Sep 16 12:08:39.396299 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.396208    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d43f24-182d-42c7-90ec-fa68f6a9ca19" containerName="discovery"
Sep 16 12:08:39.398604 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.398581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.401442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.401419    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:08:39.401442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.401434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-544wn\""
Sep 16 12:08:39.402722 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.402700    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:08:39.408883 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.408861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"]
Sep 16 12:08:39.497567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.497538    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"]
Sep 16 12:08:39.500075 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.500060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.509277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.509252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"]
Sep 16 12:08:39.537371 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.537344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.537503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.537375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmpl5\" (UniqueName: \"kubernetes.io/projected/591ec7d1-7773-4cda-bd09-32d09abf3ce1-kube-api-access-dmpl5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.537503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.537405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.594366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.594334    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"]
Sep 16 12:08:39.596837 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.596817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.605695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.605674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"]
Sep 16 12:08:39.638134 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.638107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.638261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.638137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpl5\" (UniqueName: \"kubernetes.io/projected/591ec7d1-7773-4cda-bd09-32d09abf3ce1-kube-api-access-dmpl5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.638261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.638167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.638261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.638185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.638261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.638212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.638470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.638279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tsrr\" (UniqueName: \"kubernetes.io/projected/a41d9193-d830-45c4-8699-a33cb1b27221-kube-api-access-5tsrr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.638541 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.638496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.638585 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.638539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.647161 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.647102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmpl5\" (UniqueName: \"kubernetes.io/projected/591ec7d1-7773-4cda-bd09-32d09abf3ce1-kube-api-access-dmpl5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.699163 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.699138    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"]
Sep 16 12:08:39.701933 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.701918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.707843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.707822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:39.711164 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.711137    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"]
Sep 16 12:08:39.739183 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.739155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tsrr\" (UniqueName: \"kubernetes.io/projected/a41d9193-d830-45c4-8699-a33cb1b27221-kube-api-access-5tsrr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.739325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.739193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6dz\" (UniqueName: \"kubernetes.io/projected/2d14210e-3b4e-440c-8702-712a4ef80f1d-kube-api-access-nb6dz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.739325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.739217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.739325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.739243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.739325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.739317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.739531 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.739393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.739758 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.739739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.739797 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.739750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.749250 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.749220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tsrr\" (UniqueName: \"kubernetes.io/projected/a41d9193-d830-45c4-8699-a33cb1b27221-kube-api-access-5tsrr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.810101 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.810072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:39.840781 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.840744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7n5\" (UniqueName: \"kubernetes.io/projected/1c25b9ce-a56e-49e3-8047-273418856cb4-kube-api-access-4p7n5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.840919 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.840792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.840919 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.840850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6dz\" (UniqueName: \"kubernetes.io/projected/2d14210e-3b4e-440c-8702-712a4ef80f1d-kube-api-access-nb6dz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.840919 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.840871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.840919 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.840894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.841105 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.840971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.841252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.841234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.841319 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.841300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.849403 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.849372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6dz\" (UniqueName: \"kubernetes.io/projected/2d14210e-3b4e-440c-8702-712a4ef80f1d-kube-api-access-nb6dz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.906326 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.906262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:39.942891 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.942850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.943071 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.942930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7n5\" (UniqueName: \"kubernetes.io/projected/1c25b9ce-a56e-49e3-8047-273418856cb4-kube-api-access-4p7n5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.943071 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.942966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.943310 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.943286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.943310 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.943301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.954343 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.954317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7n5\" (UniqueName: \"kubernetes.io/projected/1c25b9ce-a56e-49e3-8047-273418856cb4-kube-api-access-4p7n5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:39.981778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:39.981747    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"]
Sep 16 12:08:39.984192 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:08:39.984170    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591ec7d1_7773_4cda_bd09_32d09abf3ce1.slice/crio-06a0679642c1b29c61a7715560b3b23461a22c9667646fec374ce432ad06c575 WatchSource:0}: Error finding container 06a0679642c1b29c61a7715560b3b23461a22c9667646fec374ce432ad06c575: Status 404 returned error can't find the container with id 06a0679642c1b29c61a7715560b3b23461a22c9667646fec374ce432ad06c575
Sep 16 12:08:40.012327 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.012307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:40.080471 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.080132    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"]
Sep 16 12:08:40.083719 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:08:40.083688    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41d9193_d830_45c4_8699_a33cb1b27221.slice/crio-d22c01014a98d916d9f68c0554af5f64ac5d66859c81d232408b3c99454f470b WatchSource:0}: Error finding container d22c01014a98d916d9f68c0554af5f64ac5d66859c81d232408b3c99454f470b: Status 404 returned error can't find the container with id d22c01014a98d916d9f68c0554af5f64ac5d66859c81d232408b3c99454f470b
Sep 16 12:08:40.175390 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.175359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"]
Sep 16 12:08:40.176006 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:08:40.175983    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d14210e_3b4e_440c_8702_712a4ef80f1d.slice/crio-20c0f21c702e319cce5674b125c3072c836cef6363ffd36a46d778fc8130452e WatchSource:0}: Error finding container 20c0f21c702e319cce5674b125c3072c836cef6363ffd36a46d778fc8130452e: Status 404 returned error can't find the container with id 20c0f21c702e319cce5674b125c3072c836cef6363ffd36a46d778fc8130452e
Sep 16 12:08:40.295359 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.295334    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"]
Sep 16 12:08:40.296793 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:08:40.296764    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c25b9ce_a56e_49e3_8047_273418856cb4.slice/crio-342b8265be4e9e7db8b7b619083a2e65c78b682014d4b5548e2e70f185cd5f14 WatchSource:0}: Error finding container 342b8265be4e9e7db8b7b619083a2e65c78b682014d4b5548e2e70f185cd5f14: Status 404 returned error can't find the container with id 342b8265be4e9e7db8b7b619083a2e65c78b682014d4b5548e2e70f185cd5f14
Sep 16 12:08:40.811078 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.811046    2578 generic.go:356] "Generic (PLEG): container finished" podID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerID="dda4c8f314c539ca8077999e6e90d65f229e8fcda7b21e8199bc6698c849b713" exitCode=0
Sep 16 12:08:40.811500 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.811130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg" event={"ID":"591ec7d1-7773-4cda-bd09-32d09abf3ce1","Type":"ContainerDied","Data":"dda4c8f314c539ca8077999e6e90d65f229e8fcda7b21e8199bc6698c849b713"}
Sep 16 12:08:40.811500 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.811177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg" event={"ID":"591ec7d1-7773-4cda-bd09-32d09abf3ce1","Type":"ContainerStarted","Data":"06a0679642c1b29c61a7715560b3b23461a22c9667646fec374ce432ad06c575"}
Sep 16 12:08:40.812716 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.812698    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerID="8165f6a12e7d038b46a1bd56dcabca9e86f29beec771ea4159c8543621bc8492" exitCode=0
Sep 16 12:08:40.812813 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.812776    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6" event={"ID":"2d14210e-3b4e-440c-8702-712a4ef80f1d","Type":"ContainerDied","Data":"8165f6a12e7d038b46a1bd56dcabca9e86f29beec771ea4159c8543621bc8492"}
Sep 16 12:08:40.812875 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.812815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6" event={"ID":"2d14210e-3b4e-440c-8702-712a4ef80f1d","Type":"ContainerStarted","Data":"20c0f21c702e319cce5674b125c3072c836cef6363ffd36a46d778fc8130452e"}
Sep 16 12:08:40.814398 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.814371    2578 generic.go:356] "Generic (PLEG): container finished" podID="a41d9193-d830-45c4-8699-a33cb1b27221" containerID="4e82009665cbf0ac54003cd3f5bbf155bd5bba35c83a950ebb10c0f56cefdb9d" exitCode=0
Sep 16 12:08:40.814486 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.814460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm" event={"ID":"a41d9193-d830-45c4-8699-a33cb1b27221","Type":"ContainerDied","Data":"4e82009665cbf0ac54003cd3f5bbf155bd5bba35c83a950ebb10c0f56cefdb9d"}
Sep 16 12:08:40.814576 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.814505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm" event={"ID":"a41d9193-d830-45c4-8699-a33cb1b27221","Type":"ContainerStarted","Data":"d22c01014a98d916d9f68c0554af5f64ac5d66859c81d232408b3c99454f470b"}
Sep 16 12:08:40.816218 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.816196    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerID="5aee71ea910561115f9518311078fc4167712421fe097501a8a6540e3f2ef577" exitCode=0
Sep 16 12:08:40.816296 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.816244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs" event={"ID":"1c25b9ce-a56e-49e3-8047-273418856cb4","Type":"ContainerDied","Data":"5aee71ea910561115f9518311078fc4167712421fe097501a8a6540e3f2ef577"}
Sep 16 12:08:40.816296 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:40.816284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs" event={"ID":"1c25b9ce-a56e-49e3-8047-273418856cb4","Type":"ContainerStarted","Data":"342b8265be4e9e7db8b7b619083a2e65c78b682014d4b5548e2e70f185cd5f14"}
Sep 16 12:08:41.823072 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:41.822994    2578 generic.go:356] "Generic (PLEG): container finished" podID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerID="cdd70ac991bf4cfe5108efb11b5ea5d0b5cff01b6bea0e79d221cea138e988f1" exitCode=0
Sep 16 12:08:41.823534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:41.823080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg" event={"ID":"591ec7d1-7773-4cda-bd09-32d09abf3ce1","Type":"ContainerDied","Data":"cdd70ac991bf4cfe5108efb11b5ea5d0b5cff01b6bea0e79d221cea138e988f1"}
Sep 16 12:08:41.825050 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:41.825027    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerID="02fe904475b04f4ddc906372ccc93bbc3a15b05ed94f2d170da04b0636e0741b" exitCode=0
Sep 16 12:08:41.825171 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:41.825131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6" event={"ID":"2d14210e-3b4e-440c-8702-712a4ef80f1d","Type":"ContainerDied","Data":"02fe904475b04f4ddc906372ccc93bbc3a15b05ed94f2d170da04b0636e0741b"}
Sep 16 12:08:41.826997 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:41.826975    2578 generic.go:356] "Generic (PLEG): container finished" podID="a41d9193-d830-45c4-8699-a33cb1b27221" containerID="de21294e2d17c20e1f7bc1d8f8693e4c5dc873d296c065fb678c3a2d653c9b40" exitCode=0
Sep 16 12:08:41.827101 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:41.827053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm" event={"ID":"a41d9193-d830-45c4-8699-a33cb1b27221","Type":"ContainerDied","Data":"de21294e2d17c20e1f7bc1d8f8693e4c5dc873d296c065fb678c3a2d653c9b40"}
Sep 16 12:08:41.829213 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:41.829189    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerID="3149718eb144ccf917f0a7830764ae72a6b82bef15c47aa61b6195762f6b8918" exitCode=0
Sep 16 12:08:41.829300 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:41.829221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs" event={"ID":"1c25b9ce-a56e-49e3-8047-273418856cb4","Type":"ContainerDied","Data":"3149718eb144ccf917f0a7830764ae72a6b82bef15c47aa61b6195762f6b8918"}
Sep 16 12:08:42.836386 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:42.836351    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerID="563723a5468ff63e2ed7ccee5ac717029f9c7170ced114e093b7c150551f8bb1" exitCode=0
Sep 16 12:08:42.836797 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:42.836445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs" event={"ID":"1c25b9ce-a56e-49e3-8047-273418856cb4","Type":"ContainerDied","Data":"563723a5468ff63e2ed7ccee5ac717029f9c7170ced114e093b7c150551f8bb1"}
Sep 16 12:08:42.838592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:42.838572    2578 generic.go:356] "Generic (PLEG): container finished" podID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerID="748d9476c6dc9bcad5c40a60399e2bea8a861db1990428d2d3e1ad3c3b1c7e35" exitCode=0
Sep 16 12:08:42.838685 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:42.838665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg" event={"ID":"591ec7d1-7773-4cda-bd09-32d09abf3ce1","Type":"ContainerDied","Data":"748d9476c6dc9bcad5c40a60399e2bea8a861db1990428d2d3e1ad3c3b1c7e35"}
Sep 16 12:08:42.840792 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:42.840769    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerID="da03878ee19ecd005ffd683a4acc2d6e395439802cd87061f7b82c327d0a59a6" exitCode=0
Sep 16 12:08:42.840892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:42.840846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6" event={"ID":"2d14210e-3b4e-440c-8702-712a4ef80f1d","Type":"ContainerDied","Data":"da03878ee19ecd005ffd683a4acc2d6e395439802cd87061f7b82c327d0a59a6"}
Sep 16 12:08:42.842872 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:42.842850    2578 generic.go:356] "Generic (PLEG): container finished" podID="a41d9193-d830-45c4-8699-a33cb1b27221" containerID="16f544edbed921a215f0e60520fbd28415d0013e23422de7e8214607045515ed" exitCode=0
Sep 16 12:08:42.842950 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:42.842883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm" event={"ID":"a41d9193-d830-45c4-8699-a33cb1b27221","Type":"ContainerDied","Data":"16f544edbed921a215f0e60520fbd28415d0013e23422de7e8214607045515ed"}
Sep 16 12:08:43.987369 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:43.987341    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:44.026878 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.026855    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:44.052482 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.052459    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:44.055721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.055700    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:44.076125 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.076102    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p7n5\" (UniqueName: \"kubernetes.io/projected/1c25b9ce-a56e-49e3-8047-273418856cb4-kube-api-access-4p7n5\") pod \"1c25b9ce-a56e-49e3-8047-273418856cb4\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") "
Sep 16 12:08:44.076236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.076140    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-util\") pod \"1c25b9ce-a56e-49e3-8047-273418856cb4\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") "
Sep 16 12:08:44.076236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.076202    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-bundle\") pod \"1c25b9ce-a56e-49e3-8047-273418856cb4\" (UID: \"1c25b9ce-a56e-49e3-8047-273418856cb4\") "
Sep 16 12:08:44.076694 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.076665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-bundle" (OuterVolumeSpecName: "bundle") pod "1c25b9ce-a56e-49e3-8047-273418856cb4" (UID: "1c25b9ce-a56e-49e3-8047-273418856cb4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:44.078325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.078298    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c25b9ce-a56e-49e3-8047-273418856cb4-kube-api-access-4p7n5" (OuterVolumeSpecName: "kube-api-access-4p7n5") pod "1c25b9ce-a56e-49e3-8047-273418856cb4" (UID: "1c25b9ce-a56e-49e3-8047-273418856cb4"). InnerVolumeSpecName "kube-api-access-4p7n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:44.081622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.081599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-util" (OuterVolumeSpecName: "util") pod "1c25b9ce-a56e-49e3-8047-273418856cb4" (UID: "1c25b9ce-a56e-49e3-8047-273418856cb4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:44.177331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmpl5\" (UniqueName: \"kubernetes.io/projected/591ec7d1-7773-4cda-bd09-32d09abf3ce1-kube-api-access-dmpl5\") pod \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") "
Sep 16 12:08:44.177331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177291    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-bundle\") pod \"a41d9193-d830-45c4-8699-a33cb1b27221\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") "
Sep 16 12:08:44.177331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-bundle\") pod \"2d14210e-3b4e-440c-8702-712a4ef80f1d\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") "
Sep 16 12:08:44.177642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177345    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-util\") pod \"a41d9193-d830-45c4-8699-a33cb1b27221\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") "
Sep 16 12:08:44.177642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-util\") pod \"2d14210e-3b4e-440c-8702-712a4ef80f1d\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") "
Sep 16 12:08:44.177642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177383    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-bundle\") pod \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") "
Sep 16 12:08:44.177642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-util\") pod \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\" (UID: \"591ec7d1-7773-4cda-bd09-32d09abf3ce1\") "
Sep 16 12:08:44.177642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177424    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tsrr\" (UniqueName: \"kubernetes.io/projected/a41d9193-d830-45c4-8699-a33cb1b27221-kube-api-access-5tsrr\") pod \"a41d9193-d830-45c4-8699-a33cb1b27221\" (UID: \"a41d9193-d830-45c4-8699-a33cb1b27221\") "
Sep 16 12:08:44.177642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb6dz\" (UniqueName: \"kubernetes.io/projected/2d14210e-3b4e-440c-8702-712a4ef80f1d-kube-api-access-nb6dz\") pod \"2d14210e-3b4e-440c-8702-712a4ef80f1d\" (UID: \"2d14210e-3b4e-440c-8702-712a4ef80f1d\") "
Sep 16 12:08:44.177921 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177732    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.177921 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177752    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7n5\" (UniqueName: \"kubernetes.io/projected/1c25b9ce-a56e-49e3-8047-273418856cb4-kube-api-access-4p7n5\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.177921 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177766    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c25b9ce-a56e-49e3-8047-273418856cb4-util\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.178064 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.177985    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-bundle" (OuterVolumeSpecName: "bundle") pod "2d14210e-3b4e-440c-8702-712a4ef80f1d" (UID: "2d14210e-3b4e-440c-8702-712a4ef80f1d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:44.178728 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.178240    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-bundle" (OuterVolumeSpecName: "bundle") pod "a41d9193-d830-45c4-8699-a33cb1b27221" (UID: "a41d9193-d830-45c4-8699-a33cb1b27221"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:44.178728 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.178641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-bundle" (OuterVolumeSpecName: "bundle") pod "591ec7d1-7773-4cda-bd09-32d09abf3ce1" (UID: "591ec7d1-7773-4cda-bd09-32d09abf3ce1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:44.180142 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.180102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591ec7d1-7773-4cda-bd09-32d09abf3ce1-kube-api-access-dmpl5" (OuterVolumeSpecName: "kube-api-access-dmpl5") pod "591ec7d1-7773-4cda-bd09-32d09abf3ce1" (UID: "591ec7d1-7773-4cda-bd09-32d09abf3ce1"). InnerVolumeSpecName "kube-api-access-dmpl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:44.180224 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.180205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41d9193-d830-45c4-8699-a33cb1b27221-kube-api-access-5tsrr" (OuterVolumeSpecName: "kube-api-access-5tsrr") pod "a41d9193-d830-45c4-8699-a33cb1b27221" (UID: "a41d9193-d830-45c4-8699-a33cb1b27221"). InnerVolumeSpecName "kube-api-access-5tsrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:44.180261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.180212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d14210e-3b4e-440c-8702-712a4ef80f1d-kube-api-access-nb6dz" (OuterVolumeSpecName: "kube-api-access-nb6dz") pod "2d14210e-3b4e-440c-8702-712a4ef80f1d" (UID: "2d14210e-3b4e-440c-8702-712a4ef80f1d"). InnerVolumeSpecName "kube-api-access-nb6dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:44.183592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.183570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-util" (OuterVolumeSpecName: "util") pod "a41d9193-d830-45c4-8699-a33cb1b27221" (UID: "a41d9193-d830-45c4-8699-a33cb1b27221"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:44.183732 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.183711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-util" (OuterVolumeSpecName: "util") pod "2d14210e-3b4e-440c-8702-712a4ef80f1d" (UID: "2d14210e-3b4e-440c-8702-712a4ef80f1d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:44.184396 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.184372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-util" (OuterVolumeSpecName: "util") pod "591ec7d1-7773-4cda-bd09-32d09abf3ce1" (UID: "591ec7d1-7773-4cda-bd09-32d09abf3ce1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:08:44.279195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279142    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmpl5\" (UniqueName: \"kubernetes.io/projected/591ec7d1-7773-4cda-bd09-32d09abf3ce1-kube-api-access-dmpl5\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.279195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279190    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.279195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279201    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.279195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279209    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a41d9193-d830-45c4-8699-a33cb1b27221-util\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.279195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279217    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d14210e-3b4e-440c-8702-712a4ef80f1d-util\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.279504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279225    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.279504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279232    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/591ec7d1-7773-4cda-bd09-32d09abf3ce1-util\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.279504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279239    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tsrr\" (UniqueName: \"kubernetes.io/projected/a41d9193-d830-45c4-8699-a33cb1b27221-kube-api-access-5tsrr\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.279504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.279247    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6dz\" (UniqueName: \"kubernetes.io/projected/2d14210e-3b4e-440c-8702-712a4ef80f1d-kube-api-access-nb6dz\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:44.854636 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.854601    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6"
Sep 16 12:08:44.854636 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.854613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jgcs6" event={"ID":"2d14210e-3b4e-440c-8702-712a4ef80f1d","Type":"ContainerDied","Data":"20c0f21c702e319cce5674b125c3072c836cef6363ffd36a46d778fc8130452e"}
Sep 16 12:08:44.854895 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.854664    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c0f21c702e319cce5674b125c3072c836cef6363ffd36a46d778fc8130452e"
Sep 16 12:08:44.856682 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.856663    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm"
Sep 16 12:08:44.856827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.856662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akc8nm" event={"ID":"a41d9193-d830-45c4-8699-a33cb1b27221","Type":"ContainerDied","Data":"d22c01014a98d916d9f68c0554af5f64ac5d66859c81d232408b3c99454f470b"}
Sep 16 12:08:44.856827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.856775    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d22c01014a98d916d9f68c0554af5f64ac5d66859c81d232408b3c99454f470b"
Sep 16 12:08:44.858833 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.858807    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs"
Sep 16 12:08:44.858956 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.858835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67cxgs" event={"ID":"1c25b9ce-a56e-49e3-8047-273418856cb4","Type":"ContainerDied","Data":"342b8265be4e9e7db8b7b619083a2e65c78b682014d4b5548e2e70f185cd5f14"}
Sep 16 12:08:44.858956 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.858875    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342b8265be4e9e7db8b7b619083a2e65c78b682014d4b5548e2e70f185cd5f14"
Sep 16 12:08:44.861062 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.861039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg" event={"ID":"591ec7d1-7773-4cda-bd09-32d09abf3ce1","Type":"ContainerDied","Data":"06a0679642c1b29c61a7715560b3b23461a22c9667646fec374ce432ad06c575"}
Sep 16 12:08:44.861165 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.861087    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06a0679642c1b29c61a7715560b3b23461a22c9667646fec374ce432ad06c575"
Sep 16 12:08:44.861165 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:44.861050    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzrdg"
Sep 16 12:08:57.259770 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.259703    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76c8c8844d-t5xsw" podUID="016d9fb9-4668-4a37-bb01-9871dc33f6ce" containerName="console" containerID="cri-o://1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489" gracePeriod=15
Sep 16 12:08:57.499200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.499175    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:08:57.588038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.587947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-oauth-config\") pod \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") "
Sep 16 12:08:57.588038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.587994    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-oauth-serving-cert\") pod \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") "
Sep 16 12:08:57.588038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-trusted-ca-bundle\") pod \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") "
Sep 16 12:08:57.588038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588026    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-config\") pod \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") "
Sep 16 12:08:57.588357 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ck8g\" (UniqueName: \"kubernetes.io/projected/016d9fb9-4668-4a37-bb01-9871dc33f6ce-kube-api-access-5ck8g\") pod \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") "
Sep 16 12:08:57.588357 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-service-ca\") pod \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") "
Sep 16 12:08:57.588357 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-serving-cert\") pod \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\" (UID: \"016d9fb9-4668-4a37-bb01-9871dc33f6ce\") "
Sep 16 12:08:57.588500 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588446    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "016d9fb9-4668-4a37-bb01-9871dc33f6ce" (UID: "016d9fb9-4668-4a37-bb01-9871dc33f6ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:57.588500 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "016d9fb9-4668-4a37-bb01-9871dc33f6ce" (UID: "016d9fb9-4668-4a37-bb01-9871dc33f6ce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:57.588500 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-config" (OuterVolumeSpecName: "console-config") pod "016d9fb9-4668-4a37-bb01-9871dc33f6ce" (UID: "016d9fb9-4668-4a37-bb01-9871dc33f6ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:57.588780 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.588755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "016d9fb9-4668-4a37-bb01-9871dc33f6ce" (UID: "016d9fb9-4668-4a37-bb01-9871dc33f6ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:08:57.590381 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.590350    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "016d9fb9-4668-4a37-bb01-9871dc33f6ce" (UID: "016d9fb9-4668-4a37-bb01-9871dc33f6ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:08:57.590526 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.590493    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016d9fb9-4668-4a37-bb01-9871dc33f6ce-kube-api-access-5ck8g" (OuterVolumeSpecName: "kube-api-access-5ck8g") pod "016d9fb9-4668-4a37-bb01-9871dc33f6ce" (UID: "016d9fb9-4668-4a37-bb01-9871dc33f6ce"). InnerVolumeSpecName "kube-api-access-5ck8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:08:57.590573 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.590501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "016d9fb9-4668-4a37-bb01-9871dc33f6ce" (UID: "016d9fb9-4668-4a37-bb01-9871dc33f6ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:08:57.689264 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.689227    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-oauth-serving-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:57.689264 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.689259    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-trusted-ca-bundle\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:57.689264 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.689268    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-config\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:57.689495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.689277    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ck8g\" (UniqueName: \"kubernetes.io/projected/016d9fb9-4668-4a37-bb01-9871dc33f6ce-kube-api-access-5ck8g\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:57.689495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.689286    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/016d9fb9-4668-4a37-bb01-9871dc33f6ce-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:57.689495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.689296    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-serving-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:57.689495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.689304    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/016d9fb9-4668-4a37-bb01-9871dc33f6ce-console-oauth-config\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:08:57.922378 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.922291    2578 generic.go:356] "Generic (PLEG): container finished" podID="016d9fb9-4668-4a37-bb01-9871dc33f6ce" containerID="1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489" exitCode=0
Sep 16 12:08:57.922378 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.922336    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c8c8844d-t5xsw"
Sep 16 12:08:57.922378 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.922343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c8c8844d-t5xsw" event={"ID":"016d9fb9-4668-4a37-bb01-9871dc33f6ce","Type":"ContainerDied","Data":"1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489"}
Sep 16 12:08:57.922378 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.922378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c8c8844d-t5xsw" event={"ID":"016d9fb9-4668-4a37-bb01-9871dc33f6ce","Type":"ContainerDied","Data":"f8b060ca9435d7b62275fd4b3ae5263cdbbf52750bbcde2f04ef38ee15fadc02"}
Sep 16 12:08:57.922718 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.922397    2578 scope.go:117] "RemoveContainer" containerID="1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489"
Sep 16 12:08:57.931611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.931592    2578 scope.go:117] "RemoveContainer" containerID="1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489"
Sep 16 12:08:57.931873 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:08:57.931855    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489\": container with ID starting with 1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489 not found: ID does not exist" containerID="1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489"
Sep 16 12:08:57.931910 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.931883    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489"} err="failed to get container status \"1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489\": rpc error: code = NotFound desc = could not find container \"1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489\": container with ID starting with 1a761648c5e4fc2e25fd76ed8fac8ca9be2e942e165719339a4d6844b0650489 not found: ID does not exist"
Sep 16 12:08:57.946820 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.946798    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76c8c8844d-t5xsw"]
Sep 16 12:08:57.948612 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:57.948592    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76c8c8844d-t5xsw"]
Sep 16 12:08:58.884395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:08:58.884364    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="016d9fb9-4668-4a37-bb01-9871dc33f6ce" path="/var/lib/kubelet/pods/016d9fb9-4668-4a37-bb01-9871dc33f6ce/volumes"
Sep 16 12:09:05.779040 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779007    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"]
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779326    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41d9193-d830-45c4-8699-a33cb1b27221" containerName="util"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779339    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41d9193-d830-45c4-8699-a33cb1b27221" containerName="util"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779349    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerName="pull"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779356    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerName="pull"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779363    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerName="pull"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779368    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerName="pull"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779377    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerName="util"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779382    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerName="util"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779388    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41d9193-d830-45c4-8699-a33cb1b27221" containerName="pull"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779393    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41d9193-d830-45c4-8699-a33cb1b27221" containerName="pull"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779399    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerName="extract"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779404    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerName="extract"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerName="extract"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779417    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerName="extract"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779425    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="016d9fb9-4668-4a37-bb01-9871dc33f6ce" containerName="console"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779430    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="016d9fb9-4668-4a37-bb01-9871dc33f6ce" containerName="console"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779436    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerName="pull"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779441    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerName="pull"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779448    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerName="extract"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779453    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerName="extract"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779459    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerName="util"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerName="util"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779473    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerName="util"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779477    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerName="util"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779483    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41d9193-d830-45c4-8699-a33cb1b27221" containerName="extract"
Sep 16 12:09:05.779497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779488    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41d9193-d830-45c4-8699-a33cb1b27221" containerName="extract"
Sep 16 12:09:05.780268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779554    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41d9193-d830-45c4-8699-a33cb1b27221" containerName="extract"
Sep 16 12:09:05.780268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779564    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c25b9ce-a56e-49e3-8047-273418856cb4" containerName="extract"
Sep 16 12:09:05.780268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779571    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="591ec7d1-7773-4cda-bd09-32d09abf3ce1" containerName="extract"
Sep 16 12:09:05.780268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779577    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d14210e-3b4e-440c-8702-712a4ef80f1d" containerName="extract"
Sep 16 12:09:05.780268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.779583    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="016d9fb9-4668-4a37-bb01-9871dc33f6ce" containerName="console"
Sep 16 12:09:05.782400 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.782374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.784921 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.784898    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:09:05.785027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.784962    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 12:09:05.785027 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.784962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4k8h2\""
Sep 16 12:09:05.785955 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.785940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:09:05.786005 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.785951    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 12:09:05.789708 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.789684    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"]
Sep 16 12:09:05.846414 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.846377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.846618 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.846424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rwx\" (UniqueName: \"kubernetes.io/projected/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-kube-api-access-z2rwx\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.846618 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.846452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.946809 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.946776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.946971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.946832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.946971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.946861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rwx\" (UniqueName: \"kubernetes.io/projected/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-kube-api-access-z2rwx\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.947458 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.947437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.949367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.949337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:05.956048 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:05.956026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rwx\" (UniqueName: \"kubernetes.io/projected/8392efaa-bdc8-44a0-9fd9-d264ecb8de3d-kube-api-access-z2rwx\") pod \"kuadrant-console-plugin-fbdd4f445-xrpbn\" (UID: \"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:06.091955 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:06.091864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"
Sep 16 12:09:06.360302 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:06.360216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn"]
Sep 16 12:09:06.362968 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:09:06.362942    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8392efaa_bdc8_44a0_9fd9_d264ecb8de3d.slice/crio-d74b2aa52d7e7c074d8d3cf4e74cfc87967e175f8370fb3d11ccb2d5d76ff345 WatchSource:0}: Error finding container d74b2aa52d7e7c074d8d3cf4e74cfc87967e175f8370fb3d11ccb2d5d76ff345: Status 404 returned error can't find the container with id d74b2aa52d7e7c074d8d3cf4e74cfc87967e175f8370fb3d11ccb2d5d76ff345
Sep 16 12:09:06.966359 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:06.966323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn" event={"ID":"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d","Type":"ContainerStarted","Data":"d74b2aa52d7e7c074d8d3cf4e74cfc87967e175f8370fb3d11ccb2d5d76ff345"}
Sep 16 12:09:10.988784 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:10.988755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn" event={"ID":"8392efaa-bdc8-44a0-9fd9-d264ecb8de3d","Type":"ContainerStarted","Data":"ac799e91141db5b62df92ecabdf0fec94d371d31c5e60fdda2304716ec45798f"}
Sep 16 12:09:11.010080 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:09:11.010037    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xrpbn" podStartSLOduration=1.4951794440000001 podStartE2EDuration="6.010022111s" totalImagesPullingTime="4.514842667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:11.008373949 +0000 UTC m=+440.645698783" watchObservedRunningTime="2026-09-16 12:09:11.010022111 +0000 UTC m=+440.647346949"
Sep 16 12:10:03.752147 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:03.752106    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k5p7w"]
Sep 16 12:10:03.757765 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:03.757738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w"
Sep 16 12:10:03.760531 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:03.760490    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m64p4\""
Sep 16 12:10:03.763305 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:03.763279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k5p7w"]
Sep 16 12:10:03.880782 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:03.880746    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-d8jq2"]
Sep 16 12:10:03.882944 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:03.882927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d8jq2"
Sep 16 12:10:03.891144 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:03.891122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d8jq2"]
Sep 16 12:10:03.917849 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:03.917827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbbn\" (UniqueName: \"kubernetes.io/projected/993df87b-8d0f-4583-b23d-eb4acdb6dc8f-kube-api-access-kdbbn\") pod \"authorino-7bcfd8f9cd-k5p7w\" (UID: \"993df87b-8d0f-4583-b23d-eb4acdb6dc8f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w"
Sep 16 12:10:04.018664 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.018591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbbn\" (UniqueName: \"kubernetes.io/projected/993df87b-8d0f-4583-b23d-eb4acdb6dc8f-kube-api-access-kdbbn\") pod \"authorino-7bcfd8f9cd-k5p7w\" (UID: \"993df87b-8d0f-4583-b23d-eb4acdb6dc8f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w"
Sep 16 12:10:04.018664 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.018629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvmb\" (UniqueName: \"kubernetes.io/projected/8d717738-68bc-4449-a7f6-94bc60011f8e-kube-api-access-dwvmb\") pod \"authorino-59c974bd48-d8jq2\" (UID: \"8d717738-68bc-4449-a7f6-94bc60011f8e\") " pod="kuadrant-system/authorino-59c974bd48-d8jq2"
Sep 16 12:10:04.034552 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.034509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbbn\" (UniqueName: \"kubernetes.io/projected/993df87b-8d0f-4583-b23d-eb4acdb6dc8f-kube-api-access-kdbbn\") pod \"authorino-7bcfd8f9cd-k5p7w\" (UID: \"993df87b-8d0f-4583-b23d-eb4acdb6dc8f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w"
Sep 16 12:10:04.067325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.067300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w"
Sep 16 12:10:04.120202 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.120093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvmb\" (UniqueName: \"kubernetes.io/projected/8d717738-68bc-4449-a7f6-94bc60011f8e-kube-api-access-dwvmb\") pod \"authorino-59c974bd48-d8jq2\" (UID: \"8d717738-68bc-4449-a7f6-94bc60011f8e\") " pod="kuadrant-system/authorino-59c974bd48-d8jq2"
Sep 16 12:10:04.130633 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.130603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvmb\" (UniqueName: \"kubernetes.io/projected/8d717738-68bc-4449-a7f6-94bc60011f8e-kube-api-access-dwvmb\") pod \"authorino-59c974bd48-d8jq2\" (UID: \"8d717738-68bc-4449-a7f6-94bc60011f8e\") " pod="kuadrant-system/authorino-59c974bd48-d8jq2"
Sep 16 12:10:04.191870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.191836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d8jq2"
Sep 16 12:10:04.335196 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.335168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k5p7w"]
Sep 16 12:10:04.336468 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:10:04.336443    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod993df87b_8d0f_4583_b23d_eb4acdb6dc8f.slice/crio-67508939e773dd08fb6e50d047f2443b4d7be38c0cbb9f02646e0e5b4fe4e8d5 WatchSource:0}: Error finding container 67508939e773dd08fb6e50d047f2443b4d7be38c0cbb9f02646e0e5b4fe4e8d5: Status 404 returned error can't find the container with id 67508939e773dd08fb6e50d047f2443b4d7be38c0cbb9f02646e0e5b4fe4e8d5
Sep 16 12:10:04.458235 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:04.458207    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d8jq2"]
Sep 16 12:10:04.459968 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:10:04.459940    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d717738_68bc_4449_a7f6_94bc60011f8e.slice/crio-21271205651a1aee8ca3c9c953290274ee550d5ca0299510d207b60f181495f5 WatchSource:0}: Error finding container 21271205651a1aee8ca3c9c953290274ee550d5ca0299510d207b60f181495f5: Status 404 returned error can't find the container with id 21271205651a1aee8ca3c9c953290274ee550d5ca0299510d207b60f181495f5
Sep 16 12:10:05.259463 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:05.259428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w" event={"ID":"993df87b-8d0f-4583-b23d-eb4acdb6dc8f","Type":"ContainerStarted","Data":"67508939e773dd08fb6e50d047f2443b4d7be38c0cbb9f02646e0e5b4fe4e8d5"}
Sep 16 12:10:05.260541 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:05.260497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d8jq2" event={"ID":"8d717738-68bc-4449-a7f6-94bc60011f8e","Type":"ContainerStarted","Data":"21271205651a1aee8ca3c9c953290274ee550d5ca0299510d207b60f181495f5"}
Sep 16 12:10:08.279323 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:08.279283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w" event={"ID":"993df87b-8d0f-4583-b23d-eb4acdb6dc8f","Type":"ContainerStarted","Data":"2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c"}
Sep 16 12:10:08.280795 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:08.280768    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d8jq2" event={"ID":"8d717738-68bc-4449-a7f6-94bc60011f8e","Type":"ContainerStarted","Data":"f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9"}
Sep 16 12:10:08.295586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:08.295547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w" podStartSLOduration=2.075928137 podStartE2EDuration="5.295533285s" totalImagesPullingTime="3.219605148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:08.293500461 +0000 UTC m=+497.930825298" watchObservedRunningTime="2026-09-16 12:10:08.295533285 +0000 UTC m=+497.932858117"
Sep 16 12:10:08.308058 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:08.308013    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-d8jq2" podStartSLOduration=2.201367023 podStartE2EDuration="5.307999434s" totalImagesPullingTime="3.106632411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:08.306599555 +0000 UTC m=+497.943924410" watchObservedRunningTime="2026-09-16 12:10:08.307999434 +0000 UTC m=+497.945324272"
Sep 16 12:10:08.328367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:08.328337    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k5p7w"]
Sep 16 12:10:10.290418 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:10.290372    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w" podUID="993df87b-8d0f-4583-b23d-eb4acdb6dc8f" containerName="authorino" containerID="cri-o://2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c" gracePeriod=30
Sep 16 12:10:10.529025 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:10.529002    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w"
Sep 16 12:10:10.575827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:10.575754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdbbn\" (UniqueName: \"kubernetes.io/projected/993df87b-8d0f-4583-b23d-eb4acdb6dc8f-kube-api-access-kdbbn\") pod \"993df87b-8d0f-4583-b23d-eb4acdb6dc8f\" (UID: \"993df87b-8d0f-4583-b23d-eb4acdb6dc8f\") "
Sep 16 12:10:10.578007 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:10.577982    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993df87b-8d0f-4583-b23d-eb4acdb6dc8f-kube-api-access-kdbbn" (OuterVolumeSpecName: "kube-api-access-kdbbn") pod "993df87b-8d0f-4583-b23d-eb4acdb6dc8f" (UID: "993df87b-8d0f-4583-b23d-eb4acdb6dc8f"). InnerVolumeSpecName "kube-api-access-kdbbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:10.676405 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:10.676365    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbbn\" (UniqueName: \"kubernetes.io/projected/993df87b-8d0f-4583-b23d-eb4acdb6dc8f-kube-api-access-kdbbn\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:10:11.296368 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.296334    2578 generic.go:356] "Generic (PLEG): container finished" podID="993df87b-8d0f-4583-b23d-eb4acdb6dc8f" containerID="2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c" exitCode=0
Sep 16 12:10:11.296368 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.296383    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w"
Sep 16 12:10:11.296897 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.296409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w" event={"ID":"993df87b-8d0f-4583-b23d-eb4acdb6dc8f","Type":"ContainerDied","Data":"2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c"}
Sep 16 12:10:11.296897 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.296451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-k5p7w" event={"ID":"993df87b-8d0f-4583-b23d-eb4acdb6dc8f","Type":"ContainerDied","Data":"67508939e773dd08fb6e50d047f2443b4d7be38c0cbb9f02646e0e5b4fe4e8d5"}
Sep 16 12:10:11.296897 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.296471    2578 scope.go:117] "RemoveContainer" containerID="2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c"
Sep 16 12:10:11.305634 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.305617    2578 scope.go:117] "RemoveContainer" containerID="2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c"
Sep 16 12:10:11.305905 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:10:11.305883    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c\": container with ID starting with 2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c not found: ID does not exist" containerID="2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c"
Sep 16 12:10:11.305982 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.305912    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c"} err="failed to get container status \"2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c\": rpc error: code = NotFound desc = could not find container \"2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c\": container with ID starting with 2fcd53ca3aa7ee536f786135b3d602986b3cb3a85dbccb0ce3c6e201f7840a2c not found: ID does not exist"
Sep 16 12:10:11.312062 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.312038    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k5p7w"]
Sep 16 12:10:11.315830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:11.315813    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k5p7w"]
Sep 16 12:10:12.883499 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:12.883461    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="993df87b-8d0f-4583-b23d-eb4acdb6dc8f" path="/var/lib/kubelet/pods/993df87b-8d0f-4583-b23d-eb4acdb6dc8f/volumes"
Sep 16 12:10:31.409384 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.409344    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-kglx9"]
Sep 16 12:10:31.409812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.409689    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="993df87b-8d0f-4583-b23d-eb4acdb6dc8f" containerName="authorino"
Sep 16 12:10:31.409812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.409702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="993df87b-8d0f-4583-b23d-eb4acdb6dc8f" containerName="authorino"
Sep 16 12:10:31.409812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.409758    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="993df87b-8d0f-4583-b23d-eb4acdb6dc8f" containerName="authorino"
Sep 16 12:10:31.412940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.412918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-kglx9"
Sep 16 12:10:31.420240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.420216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-kglx9"]
Sep 16 12:10:31.435657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.435630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5969w\" (UniqueName: \"kubernetes.io/projected/f61f7556-000e-43d7-9b41-79d2a2991007-kube-api-access-5969w\") pod \"authorino-557f876f69-kglx9\" (UID: \"f61f7556-000e-43d7-9b41-79d2a2991007\") " pod="kuadrant-system/authorino-557f876f69-kglx9"
Sep 16 12:10:31.536026 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.535996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5969w\" (UniqueName: \"kubernetes.io/projected/f61f7556-000e-43d7-9b41-79d2a2991007-kube-api-access-5969w\") pod \"authorino-557f876f69-kglx9\" (UID: \"f61f7556-000e-43d7-9b41-79d2a2991007\") " pod="kuadrant-system/authorino-557f876f69-kglx9"
Sep 16 12:10:31.544408 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.544375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5969w\" (UniqueName: \"kubernetes.io/projected/f61f7556-000e-43d7-9b41-79d2a2991007-kube-api-access-5969w\") pod \"authorino-557f876f69-kglx9\" (UID: \"f61f7556-000e-43d7-9b41-79d2a2991007\") " pod="kuadrant-system/authorino-557f876f69-kglx9"
Sep 16 12:10:31.670109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.670018    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-kglx9"]
Sep 16 12:10:31.670304 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.670287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-kglx9"
Sep 16 12:10:31.696969 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.696939    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-859945ff7d-54vql"]
Sep 16 12:10:31.705418 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.705396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-859945ff7d-54vql"
Sep 16 12:10:31.713442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.713394    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-859945ff7d-54vql"]
Sep 16 12:10:31.730999 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.730973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-859945ff7d-54vql"]
Sep 16 12:10:31.731215 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:10:31.731198    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-km68f], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-859945ff7d-54vql" podUID="ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f"
Sep 16 12:10:31.737272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.737243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km68f\" (UniqueName: \"kubernetes.io/projected/ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f-kube-api-access-km68f\") pod \"authorino-859945ff7d-54vql\" (UID: \"ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f\") " pod="kuadrant-system/authorino-859945ff7d-54vql"
Sep 16 12:10:31.754602 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.754570    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f88d889d4-nr7hn"]
Sep 16 12:10:31.759003 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.758637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:10:31.762614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.761332    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 12:10:31.763770 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.763749    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f88d889d4-nr7hn"]
Sep 16 12:10:31.837802 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.837773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km68f\" (UniqueName: \"kubernetes.io/projected/ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f-kube-api-access-km68f\") pod \"authorino-859945ff7d-54vql\" (UID: \"ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f\") " pod="kuadrant-system/authorino-859945ff7d-54vql"
Sep 16 12:10:31.837981 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.837824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-tls-cert\") pod \"authorino-f88d889d4-nr7hn\" (UID: \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\") " pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:10:31.837981 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.837951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rllxm\" (UniqueName: \"kubernetes.io/projected/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-kube-api-access-rllxm\") pod \"authorino-f88d889d4-nr7hn\" (UID: \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\") " pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:10:31.846037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.846012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km68f\" (UniqueName: \"kubernetes.io/projected/ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f-kube-api-access-km68f\") pod \"authorino-859945ff7d-54vql\" (UID: \"ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f\") " pod="kuadrant-system/authorino-859945ff7d-54vql"
Sep 16 12:10:31.938664 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.938575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-tls-cert\") pod \"authorino-f88d889d4-nr7hn\" (UID: \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\") " pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:10:31.938664 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.938630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rllxm\" (UniqueName: \"kubernetes.io/projected/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-kube-api-access-rllxm\") pod \"authorino-f88d889d4-nr7hn\" (UID: \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\") " pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:10:31.941282 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.941253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-tls-cert\") pod \"authorino-f88d889d4-nr7hn\" (UID: \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\") " pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:10:31.948308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.948287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rllxm\" (UniqueName: \"kubernetes.io/projected/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-kube-api-access-rllxm\") pod \"authorino-f88d889d4-nr7hn\" (UID: \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\") " pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:10:31.958553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:31.956630    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-kglx9"]
Sep 16 12:10:32.070692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.070659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:10:32.340809 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.340782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f88d889d4-nr7hn"]
Sep 16 12:10:32.342817 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:10:32.342792    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36a598fc_5a9b_4daa_89e0_2c3e8b6b891c.slice/crio-4547e8a3e6618a9bf615caa8e6f885716a2a9e55ba727518deaa13149b1425ee WatchSource:0}: Error finding container 4547e8a3e6618a9bf615caa8e6f885716a2a9e55ba727518deaa13149b1425ee: Status 404 returned error can't find the container with id 4547e8a3e6618a9bf615caa8e6f885716a2a9e55ba727518deaa13149b1425ee
Sep 16 12:10:32.402058 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.402027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f88d889d4-nr7hn" event={"ID":"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c","Type":"ContainerStarted","Data":"4547e8a3e6618a9bf615caa8e6f885716a2a9e55ba727518deaa13149b1425ee"}
Sep 16 12:10:32.403259 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.403228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-kglx9" event={"ID":"f61f7556-000e-43d7-9b41-79d2a2991007","Type":"ContainerStarted","Data":"c1e41a66c0043a3635ce392b3693819c846ec6cf0c845dfb40bf59a61d9dd934"}
Sep 16 12:10:32.403412 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.403277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-859945ff7d-54vql"
Sep 16 12:10:32.411392 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.411373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-859945ff7d-54vql"
Sep 16 12:10:32.443184 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.443157    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km68f\" (UniqueName: \"kubernetes.io/projected/ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f-kube-api-access-km68f\") pod \"ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f\" (UID: \"ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f\") "
Sep 16 12:10:32.445333 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.445307    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f-kube-api-access-km68f" (OuterVolumeSpecName: "kube-api-access-km68f") pod "ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f" (UID: "ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f"). InnerVolumeSpecName "kube-api-access-km68f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:32.543677 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:32.543653    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km68f\" (UniqueName: \"kubernetes.io/projected/ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f-kube-api-access-km68f\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:10:33.410066 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.410027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f88d889d4-nr7hn" event={"ID":"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c","Type":"ContainerStarted","Data":"a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be"}
Sep 16 12:10:33.411439 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.411419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-859945ff7d-54vql"
Sep 16 12:10:33.411858 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.411463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-kglx9" event={"ID":"f61f7556-000e-43d7-9b41-79d2a2991007","Type":"ContainerStarted","Data":"5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e"}
Sep 16 12:10:33.411858 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.411466    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-kglx9" podUID="f61f7556-000e-43d7-9b41-79d2a2991007" containerName="authorino" containerID="cri-o://5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e" gracePeriod=30
Sep 16 12:10:33.427653 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.427612    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f88d889d4-nr7hn" podStartSLOduration=2.085170168 podStartE2EDuration="2.427598301s" totalImagesPullingTime="342.428133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:33.425595075 +0000 UTC m=+523.062919913" watchObservedRunningTime="2026-09-16 12:10:33.427598301 +0000 UTC m=+523.064923130"
Sep 16 12:10:33.441021 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.440981    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-kglx9" podStartSLOduration=1.5641317940000001 podStartE2EDuration="2.440968283s" totalImagesPullingTime="876.836489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:33.439415491 +0000 UTC m=+523.076740330" watchObservedRunningTime="2026-09-16 12:10:33.440968283 +0000 UTC m=+523.078293120"
Sep 16 12:10:33.453173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.453146    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d8jq2"]
Sep 16 12:10:33.453345 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.453323    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-d8jq2" podUID="8d717738-68bc-4449-a7f6-94bc60011f8e" containerName="authorino" containerID="cri-o://f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9" gracePeriod=30
Sep 16 12:10:33.468459 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.468433    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-859945ff7d-54vql"]
Sep 16 12:10:33.473956 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.473932    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-859945ff7d-54vql"]
Sep 16 12:10:33.714272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.714247    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-kglx9"
Sep 16 12:10:33.716970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.716953    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d8jq2"
Sep 16 12:10:33.753844 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.753821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5969w\" (UniqueName: \"kubernetes.io/projected/f61f7556-000e-43d7-9b41-79d2a2991007-kube-api-access-5969w\") pod \"f61f7556-000e-43d7-9b41-79d2a2991007\" (UID: \"f61f7556-000e-43d7-9b41-79d2a2991007\") "
Sep 16 12:10:33.754021 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.753880    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwvmb\" (UniqueName: \"kubernetes.io/projected/8d717738-68bc-4449-a7f6-94bc60011f8e-kube-api-access-dwvmb\") pod \"8d717738-68bc-4449-a7f6-94bc60011f8e\" (UID: \"8d717738-68bc-4449-a7f6-94bc60011f8e\") "
Sep 16 12:10:33.756109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.756082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61f7556-000e-43d7-9b41-79d2a2991007-kube-api-access-5969w" (OuterVolumeSpecName: "kube-api-access-5969w") pod "f61f7556-000e-43d7-9b41-79d2a2991007" (UID: "f61f7556-000e-43d7-9b41-79d2a2991007"). InnerVolumeSpecName "kube-api-access-5969w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:33.756207 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.756105    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d717738-68bc-4449-a7f6-94bc60011f8e-kube-api-access-dwvmb" (OuterVolumeSpecName: "kube-api-access-dwvmb") pod "8d717738-68bc-4449-a7f6-94bc60011f8e" (UID: "8d717738-68bc-4449-a7f6-94bc60011f8e"). InnerVolumeSpecName "kube-api-access-dwvmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:33.854684 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.854637    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5969w\" (UniqueName: \"kubernetes.io/projected/f61f7556-000e-43d7-9b41-79d2a2991007-kube-api-access-5969w\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:10:33.854684 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:33.854677    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwvmb\" (UniqueName: \"kubernetes.io/projected/8d717738-68bc-4449-a7f6-94bc60011f8e-kube-api-access-dwvmb\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:10:34.417829 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.417799    2578 generic.go:356] "Generic (PLEG): container finished" podID="f61f7556-000e-43d7-9b41-79d2a2991007" containerID="5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e" exitCode=0
Sep 16 12:10:34.418261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.417839    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-kglx9"
Sep 16 12:10:34.418261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.417880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-kglx9" event={"ID":"f61f7556-000e-43d7-9b41-79d2a2991007","Type":"ContainerDied","Data":"5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e"}
Sep 16 12:10:34.418261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.417929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-kglx9" event={"ID":"f61f7556-000e-43d7-9b41-79d2a2991007","Type":"ContainerDied","Data":"c1e41a66c0043a3635ce392b3693819c846ec6cf0c845dfb40bf59a61d9dd934"}
Sep 16 12:10:34.418261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.417950    2578 scope.go:117] "RemoveContainer" containerID="5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e"
Sep 16 12:10:34.419283 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.419259    2578 generic.go:356] "Generic (PLEG): container finished" podID="8d717738-68bc-4449-a7f6-94bc60011f8e" containerID="f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9" exitCode=0
Sep 16 12:10:34.419366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.419294    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d8jq2"
Sep 16 12:10:34.419366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.419333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d8jq2" event={"ID":"8d717738-68bc-4449-a7f6-94bc60011f8e","Type":"ContainerDied","Data":"f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9"}
Sep 16 12:10:34.419366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.419360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d8jq2" event={"ID":"8d717738-68bc-4449-a7f6-94bc60011f8e","Type":"ContainerDied","Data":"21271205651a1aee8ca3c9c953290274ee550d5ca0299510d207b60f181495f5"}
Sep 16 12:10:34.428250 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.428231    2578 scope.go:117] "RemoveContainer" containerID="5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e"
Sep 16 12:10:34.428578 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:10:34.428527    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e\": container with ID starting with 5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e not found: ID does not exist" containerID="5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e"
Sep 16 12:10:34.428578 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.428560    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e"} err="failed to get container status \"5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e\": rpc error: code = NotFound desc = could not find container \"5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e\": container with ID starting with 5a93fddbaca024881b08c3a2b9eff586ef7f3fc6d2740dc5b1cc12ac9503748e not found: ID does not exist"
Sep 16 12:10:34.428680 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.428582    2578 scope.go:117] "RemoveContainer" containerID="f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9"
Sep 16 12:10:34.436397 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.436377    2578 scope.go:117] "RemoveContainer" containerID="f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9"
Sep 16 12:10:34.436678 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:10:34.436638    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9\": container with ID starting with f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9 not found: ID does not exist" containerID="f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9"
Sep 16 12:10:34.436752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.436687    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9"} err="failed to get container status \"f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9\": rpc error: code = NotFound desc = could not find container \"f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9\": container with ID starting with f2e499f9c1aaab1fed701bc9391a98a44efd3f63c948f61ebf1dee3d7a2440a9 not found: ID does not exist"
Sep 16 12:10:34.443451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.443431    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d8jq2"]
Sep 16 12:10:34.447058 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.447038    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d8jq2"]
Sep 16 12:10:34.461317 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.459167    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-kglx9"]
Sep 16 12:10:34.463466 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.463442    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-kglx9"]
Sep 16 12:10:34.883812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.883782    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d717738-68bc-4449-a7f6-94bc60011f8e" path="/var/lib/kubelet/pods/8d717738-68bc-4449-a7f6-94bc60011f8e/volumes"
Sep 16 12:10:34.884086 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.884074    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f" path="/var/lib/kubelet/pods/ea2b8016-2738-4ce4-80db-5a4aa2cf1c9f/volumes"
Sep 16 12:10:34.884258 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:34.884248    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f61f7556-000e-43d7-9b41-79d2a2991007" path="/var/lib/kubelet/pods/f61f7556-000e-43d7-9b41-79d2a2991007/volumes"
Sep 16 12:10:38.325833 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.325798    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d64f4dd7b-r4njc"]
Sep 16 12:10:38.326316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.326239    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f61f7556-000e-43d7-9b41-79d2a2991007" containerName="authorino"
Sep 16 12:10:38.326316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.326257    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f61f7556-000e-43d7-9b41-79d2a2991007" containerName="authorino"
Sep 16 12:10:38.326316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.326290    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d717738-68bc-4449-a7f6-94bc60011f8e" containerName="authorino"
Sep 16 12:10:38.326316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.326298    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d717738-68bc-4449-a7f6-94bc60011f8e" containerName="authorino"
Sep 16 12:10:38.326551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.326380    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f61f7556-000e-43d7-9b41-79d2a2991007" containerName="authorino"
Sep 16 12:10:38.326551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.326395    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d717738-68bc-4449-a7f6-94bc60011f8e" containerName="authorino"
Sep 16 12:10:38.331112 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.331086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:38.334001 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.333970    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 12:10:38.334160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.334141    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddsz6\""
Sep 16 12:10:38.334232 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.334167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 12:10:38.335474 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.335458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 12:10:38.341958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.341932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d64f4dd7b-r4njc"]
Sep 16 12:10:38.396230 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.396190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwk2\" (UniqueName: \"kubernetes.io/projected/164c8f95-75a8-4ada-ad30-5a12a462515f-kube-api-access-mcwk2\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:38.396230 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.396235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:38.396446 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.396252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-metrics-certs\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:38.497268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.497234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwk2\" (UniqueName: \"kubernetes.io/projected/164c8f95-75a8-4ada-ad30-5a12a462515f-kube-api-access-mcwk2\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:38.497440 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.497275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:38.497440 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.497302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-metrics-certs\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:38.497440 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:10:38.497397    2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 16 12:10:38.497583 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:10:38.497465    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert podName:164c8f95-75a8-4ada-ad30-5a12a462515f nodeName:}" failed. No retries permitted until 2026-09-16 12:10:38.997446078 +0000 UTC m=+528.634770894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert") pod "maas-controller-d64f4dd7b-r4njc" (UID: "164c8f95-75a8-4ada-ad30-5a12a462515f") : secret "maas-controller-webhook-cert" not found
Sep 16 12:10:38.499897 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.499878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-metrics-certs\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:38.507800 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:38.507777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwk2\" (UniqueName: \"kubernetes.io/projected/164c8f95-75a8-4ada-ad30-5a12a462515f-kube-api-access-mcwk2\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:39.001645 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:39.001588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:39.004165 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:39.004137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert\") pod \"maas-controller-d64f4dd7b-r4njc\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") " pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:39.241622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:39.241588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:39.518985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:39.517036    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d64f4dd7b-r4njc"]
Sep 16 12:10:39.521048 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:10:39.521010    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164c8f95_75a8_4ada_ad30_5a12a462515f.slice/crio-028a024315b1183cdeccffe3936e381d0af8678006affa1ef4e35f4292805621 WatchSource:0}: Error finding container 028a024315b1183cdeccffe3936e381d0af8678006affa1ef4e35f4292805621: Status 404 returned error can't find the container with id 028a024315b1183cdeccffe3936e381d0af8678006affa1ef4e35f4292805621
Sep 16 12:10:40.455274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:40.455240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d64f4dd7b-r4njc" event={"ID":"164c8f95-75a8-4ada-ad30-5a12a462515f","Type":"ContainerStarted","Data":"028a024315b1183cdeccffe3936e381d0af8678006affa1ef4e35f4292805621"}
Sep 16 12:10:42.468484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:42.468450    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d64f4dd7b-r4njc" event={"ID":"164c8f95-75a8-4ada-ad30-5a12a462515f","Type":"ContainerStarted","Data":"408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e"}
Sep 16 12:10:42.468955 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:42.468613    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:42.486434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:42.486380    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d64f4dd7b-r4njc" podStartSLOduration=2.036658225 podStartE2EDuration="4.486363169s" totalImagesPullingTime="2.449704944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:42.484163655 +0000 UTC m=+532.121488495" watchObservedRunningTime="2026-09-16 12:10:42.486363169 +0000 UTC m=+532.123688006"
Sep 16 12:10:53.475320 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:53.475293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:53.911830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:53.911750    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d64f4dd7b-r4njc"]
Sep 16 12:10:53.912003 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:53.911968    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d64f4dd7b-r4njc" podUID="164c8f95-75a8-4ada-ad30-5a12a462515f" containerName="manager" containerID="cri-o://408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e" gracePeriod=30
Sep 16 12:10:54.153816 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.153792    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:54.230443 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.230414    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert\") pod \"164c8f95-75a8-4ada-ad30-5a12a462515f\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") "
Sep 16 12:10:54.230620 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.230457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcwk2\" (UniqueName: \"kubernetes.io/projected/164c8f95-75a8-4ada-ad30-5a12a462515f-kube-api-access-mcwk2\") pod \"164c8f95-75a8-4ada-ad30-5a12a462515f\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") "
Sep 16 12:10:54.230620 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.230477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-metrics-certs\") pod \"164c8f95-75a8-4ada-ad30-5a12a462515f\" (UID: \"164c8f95-75a8-4ada-ad30-5a12a462515f\") "
Sep 16 12:10:54.232951 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.232920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164c8f95-75a8-4ada-ad30-5a12a462515f-kube-api-access-mcwk2" (OuterVolumeSpecName: "kube-api-access-mcwk2") pod "164c8f95-75a8-4ada-ad30-5a12a462515f" (UID: "164c8f95-75a8-4ada-ad30-5a12a462515f"). InnerVolumeSpecName "kube-api-access-mcwk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:54.233076 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.232943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "164c8f95-75a8-4ada-ad30-5a12a462515f" (UID: "164c8f95-75a8-4ada-ad30-5a12a462515f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:54.233076 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.232958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "164c8f95-75a8-4ada-ad30-5a12a462515f" (UID: "164c8f95-75a8-4ada-ad30-5a12a462515f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:54.331844 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.331808    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwk2\" (UniqueName: \"kubernetes.io/projected/164c8f95-75a8-4ada-ad30-5a12a462515f-kube-api-access-mcwk2\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:10:54.331844 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.331839    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:10:54.331844 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.331848    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164c8f95-75a8-4ada-ad30-5a12a462515f-webhook-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:10:54.528332 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.528239    2578 generic.go:356] "Generic (PLEG): container finished" podID="164c8f95-75a8-4ada-ad30-5a12a462515f" containerID="408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e" exitCode=0
Sep 16 12:10:54.528332 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.528296    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d64f4dd7b-r4njc"
Sep 16 12:10:54.528332 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.528316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d64f4dd7b-r4njc" event={"ID":"164c8f95-75a8-4ada-ad30-5a12a462515f","Type":"ContainerDied","Data":"408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e"}
Sep 16 12:10:54.528860 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.528368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d64f4dd7b-r4njc" event={"ID":"164c8f95-75a8-4ada-ad30-5a12a462515f","Type":"ContainerDied","Data":"028a024315b1183cdeccffe3936e381d0af8678006affa1ef4e35f4292805621"}
Sep 16 12:10:54.528860 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.528398    2578 scope.go:117] "RemoveContainer" containerID="408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e"
Sep 16 12:10:54.538266 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.538246    2578 scope.go:117] "RemoveContainer" containerID="408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e"
Sep 16 12:10:54.538557 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:10:54.538528    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e\": container with ID starting with 408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e not found: ID does not exist" containerID="408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e"
Sep 16 12:10:54.538659 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.538568    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e"} err="failed to get container status \"408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e\": rpc error: code = NotFound desc = could not find container \"408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e\": container with ID starting with 408d0f1e429fc2e552cbbb370305d7bb916b21a41e1c8c8ea5368a658bdb703e not found: ID does not exist"
Sep 16 12:10:54.550834 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.550806    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d64f4dd7b-r4njc"]
Sep 16 12:10:54.554821 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.554798    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d64f4dd7b-r4njc"]
Sep 16 12:10:54.885238 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:10:54.885156    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="164c8f95-75a8-4ada-ad30-5a12a462515f" path="/var/lib/kubelet/pods/164c8f95-75a8-4ada-ad30-5a12a462515f/volumes"
Sep 16 12:11:16.224038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.224003    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"]
Sep 16 12:11:16.224546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.224299    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="164c8f95-75a8-4ada-ad30-5a12a462515f" containerName="manager"
Sep 16 12:11:16.224546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.224310    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="164c8f95-75a8-4ada-ad30-5a12a462515f" containerName="manager"
Sep 16 12:11:16.224546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.224361    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="164c8f95-75a8-4ada-ad30-5a12a462515f" containerName="manager"
Sep 16 12:11:16.228494 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.228478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.231021 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.230995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 12:11:16.231152 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.231084    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:11:16.231152 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.231118    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:11:16.232020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.231996    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7fbn\""
Sep 16 12:11:16.232141 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.232044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:11:16.236195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.235990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"]
Sep 16 12:11:16.405210 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.405178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-maas-api-tls\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.405391 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.405314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmm6\" (UniqueName: \"kubernetes.io/projected/d5788b24-c547-42f5-8774-c5c8306065c5-kube-api-access-xgmm6\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.405469 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.405445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5788b24-c547-42f5-8774-c5c8306065c5-openshift-service-ca\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.405553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.405493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-metrics-certs\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.506041 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.505964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-maas-api-tls\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.506041 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.505995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmm6\" (UniqueName: \"kubernetes.io/projected/d5788b24-c547-42f5-8774-c5c8306065c5-kube-api-access-xgmm6\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.506238 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.506044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5788b24-c547-42f5-8774-c5c8306065c5-openshift-service-ca\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.506238 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.506064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-metrics-certs\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.506779 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.506757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5788b24-c547-42f5-8774-c5c8306065c5-openshift-service-ca\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.508896 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.508781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-metrics-certs\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.509031 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.508980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-maas-api-tls\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.517019 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.516991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmm6\" (UniqueName: \"kubernetes.io/projected/d5788b24-c547-42f5-8774-c5c8306065c5-kube-api-access-xgmm6\") pod \"maas-api-6cd6bf8c7c-jbdg5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.539856 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.539835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:16.811385 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:16.811356    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"]
Sep 16 12:11:16.812712 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:11:16.812686    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5788b24_c547_42f5_8774_c5c8306065c5.slice/crio-05a130c3ea9553df4c7e74f4bb8761b406b1964cdffef08bc31198fe0eb68b3c WatchSource:0}: Error finding container 05a130c3ea9553df4c7e74f4bb8761b406b1964cdffef08bc31198fe0eb68b3c: Status 404 returned error can't find the container with id 05a130c3ea9553df4c7e74f4bb8761b406b1964cdffef08bc31198fe0eb68b3c
Sep 16 12:11:17.649871 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:17.649833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5" event={"ID":"d5788b24-c547-42f5-8774-c5c8306065c5","Type":"ContainerStarted","Data":"05a130c3ea9553df4c7e74f4bb8761b406b1964cdffef08bc31198fe0eb68b3c"}
Sep 16 12:11:18.656177 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:18.656086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5" event={"ID":"d5788b24-c547-42f5-8774-c5c8306065c5","Type":"ContainerStarted","Data":"dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec"}
Sep 16 12:11:18.656644 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:18.656185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:18.673866 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:18.673817    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5" podStartSLOduration=1.192493067 podStartE2EDuration="2.673801136s" totalImagesPullingTime="1.481308069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:18.671253571 +0000 UTC m=+568.308578408" watchObservedRunningTime="2026-09-16 12:11:18.673801136 +0000 UTC m=+568.311125973"
Sep 16 12:11:24.665647 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:24.665615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:11:39.353865 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.353822    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"]
Sep 16 12:11:39.356404 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.356384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.358857 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.358836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 12:11:39.360033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.360016    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 12:11:39.360033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.360026    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5j28l\""
Sep 16 12:11:39.360190 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.360085    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 12:11:39.366335 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.366316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"]
Sep 16 12:11:39.385817 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.385792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmht\" (UniqueName: \"kubernetes.io/projected/f768c5bc-d430-487d-8052-7920af9aa205-kube-api-access-spmht\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.385908 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.385839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.385908 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.385868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f768c5bc-d430-487d-8052-7920af9aa205-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.385982 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.385920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.385982 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.385966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.386044 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.386008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmht\" (UniqueName: \"kubernetes.io/projected/f768c5bc-d430-487d-8052-7920af9aa205-kube-api-access-spmht\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f768c5bc-d430-487d-8052-7920af9aa205-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486785 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.486892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.486850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.488942 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.488912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f768c5bc-d430-487d-8052-7920af9aa205-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.489108 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.489089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f768c5bc-d430-487d-8052-7920af9aa205-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.494767 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.494748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmht\" (UniqueName: \"kubernetes.io/projected/f768c5bc-d430-487d-8052-7920af9aa205-kube-api-access-spmht\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht\" (UID: \"f768c5bc-d430-487d-8052-7920af9aa205\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.667039 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.666956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:39.934645 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:39.934617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"]
Sep 16 12:11:39.936278 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:11:39.936252    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf768c5bc_d430_487d_8052_7920af9aa205.slice/crio-368513242e3ce4db6dba4615ffa0905be77d4537cf79328b8ccd7e292e87539e WatchSource:0}: Error finding container 368513242e3ce4db6dba4615ffa0905be77d4537cf79328b8ccd7e292e87539e: Status 404 returned error can't find the container with id 368513242e3ce4db6dba4615ffa0905be77d4537cf79328b8ccd7e292e87539e
Sep 16 12:11:40.763243 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:40.763195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht" event={"ID":"f768c5bc-d430-487d-8052-7920af9aa205","Type":"ContainerStarted","Data":"368513242e3ce4db6dba4615ffa0905be77d4537cf79328b8ccd7e292e87539e"}
Sep 16 12:11:41.769132 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:41.769097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht" event={"ID":"f768c5bc-d430-487d-8052-7920af9aa205","Type":"ContainerStarted","Data":"b9ba6d145acb6f9e89f2768ad2a62ba8ff79dd50f14e31fd4edbdd682798a3b7"}
Sep 16 12:11:41.793251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:41.793205    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht" podStartSLOduration=1.8299034 podStartE2EDuration="2.793190126s" totalImagesPullingTime="963.286726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:41.790980792 +0000 UTC m=+591.428305634" watchObservedRunningTime="2026-09-16 12:11:41.793190126 +0000 UTC m=+591.430514963"
Sep 16 12:11:49.668081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:49.668032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:49.668559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:49.668174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:49.680922 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:49.680893    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:49.823322 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:49.823290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht"
Sep 16 12:11:50.819863 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:50.819805    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:11:50.821078 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:50.821059    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:11:55.548188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.548152    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"]
Sep 16 12:11:55.550249 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.550233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.552996 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.552977    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 12:11:55.562207 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.562186    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"]
Sep 16 12:11:55.620051 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.619948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.620317 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.620296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.620444 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.620430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.620621 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.620607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.620749 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.620735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8n97\" (UniqueName: \"kubernetes.io/projected/d25310be-d0cc-412a-881f-657aef923a53-kube-api-access-m8n97\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.621014 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.620995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25310be-d0cc-412a-881f-657aef923a53-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.721507 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.721469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.721705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.721537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8n97\" (UniqueName: \"kubernetes.io/projected/d25310be-d0cc-412a-881f-657aef923a53-kube-api-access-m8n97\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.721705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.721571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25310be-d0cc-412a-881f-657aef923a53-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.721705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.721618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.721705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.721654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.721705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.721684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.721956 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.721881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.722022 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.721972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.722077 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.722059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.723997 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.723973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25310be-d0cc-412a-881f-657aef923a53-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.724370 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.724347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25310be-d0cc-412a-881f-657aef923a53-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.729746 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.729726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8n97\" (UniqueName: \"kubernetes.io/projected/d25310be-d0cc-412a-881f-657aef923a53-kube-api-access-m8n97\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8\" (UID: \"d25310be-d0cc-412a-881f-657aef923a53\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:55.860718 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:55.860635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:11:56.131991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:56.131963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"]
Sep 16 12:11:56.133926 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:11:56.133903    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25310be_d0cc_412a_881f_657aef923a53.slice/crio-cf5f87ab9cc2febee7c073a190f5c1b74adeb71e9943e3dda034b60dff23d936 WatchSource:0}: Error finding container cf5f87ab9cc2febee7c073a190f5c1b74adeb71e9943e3dda034b60dff23d936: Status 404 returned error can't find the container with id cf5f87ab9cc2febee7c073a190f5c1b74adeb71e9943e3dda034b60dff23d936
Sep 16 12:11:56.856672 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:56.856623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8" event={"ID":"d25310be-d0cc-412a-881f-657aef923a53","Type":"ContainerStarted","Data":"f5733c525bb38ff73060af33f4d7bea7ba97611c1be3e34131b19ffdeb4d1aa3"}
Sep 16 12:11:56.857177 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:56.856700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8" event={"ID":"d25310be-d0cc-412a-881f-657aef923a53","Type":"ContainerStarted","Data":"cf5f87ab9cc2febee7c073a190f5c1b74adeb71e9943e3dda034b60dff23d936"}
Sep 16 12:11:56.877711 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:11:56.877640    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8" podStartSLOduration=1.710208497 podStartE2EDuration="1.877619281s" totalImagesPullingTime="167.410784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:56.874166952 +0000 UTC m=+606.511491790" watchObservedRunningTime="2026-09-16 12:11:56.877619281 +0000 UTC m=+606.514944121"
Sep 16 12:12:05.861262 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:05.861225    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:12:05.861714 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:05.861323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:12:05.874615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:05.874590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:12:05.913785 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:05.913758    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8"
Sep 16 12:12:44.957739 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:44.957708    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"]
Sep 16 12:12:44.962707 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:44.962678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:44.967696 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:44.967667    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 12:12:44.970455 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:44.970434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"]
Sep 16 12:12:45.047599 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.047566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.047979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.047601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.047979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.047632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmjq\" (UniqueName: \"kubernetes.io/projected/e8b212f8-e066-4ea3-b033-ea1e503e8188-kube-api-access-bwmjq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.047979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.047673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b212f8-e066-4ea3-b033-ea1e503e8188-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.047979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.047716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.047979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.047747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149159 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmjq\" (UniqueName: \"kubernetes.io/projected/e8b212f8-e066-4ea3-b033-ea1e503e8188-kube-api-access-bwmjq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b212f8-e066-4ea3-b033-ea1e503e8188-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149724 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149724 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.149819 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.149739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.151600 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.151572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b212f8-e066-4ea3-b033-ea1e503e8188-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.151856 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.151838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b212f8-e066-4ea3-b033-ea1e503e8188-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.157217 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.157198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmjq\" (UniqueName: \"kubernetes.io/projected/e8b212f8-e066-4ea3-b033-ea1e503e8188-kube-api-access-bwmjq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw\" (UID: \"e8b212f8-e066-4ea3-b033-ea1e503e8188\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.274405 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.274318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:45.546527 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.546490    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"]
Sep 16 12:12:45.547711 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:12:45.547687    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8b212f8_e066_4ea3_b033_ea1e503e8188.slice/crio-28a9f675be226b1a91164151609414e552052ba377f451392b95e9df2641a94d WatchSource:0}: Error finding container 28a9f675be226b1a91164151609414e552052ba377f451392b95e9df2641a94d: Status 404 returned error can't find the container with id 28a9f675be226b1a91164151609414e552052ba377f451392b95e9df2641a94d
Sep 16 12:12:45.549382 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:45.549364    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:12:46.100173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:46.100076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw" event={"ID":"e8b212f8-e066-4ea3-b033-ea1e503e8188","Type":"ContainerStarted","Data":"61ff927027bdfb89acdee0acb71586bdc81558a90f532d2bf398fa5bbde4919f"}
Sep 16 12:12:46.100173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:46.100124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw" event={"ID":"e8b212f8-e066-4ea3-b033-ea1e503e8188","Type":"ContainerStarted","Data":"28a9f675be226b1a91164151609414e552052ba377f451392b95e9df2641a94d"}
Sep 16 12:12:46.118928 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:46.118879    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw" podStartSLOduration=1.920994736 podStartE2EDuration="2.118864031s" totalImagesPullingTime="197.869295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:46.116808113 +0000 UTC m=+655.754132964" watchObservedRunningTime="2026-09-16 12:12:46.118864031 +0000 UTC m=+655.756188906"
Sep 16 12:12:55.275388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:55.275355    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:55.275877 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:55.275503    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:55.288380 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:55.288347    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:12:56.158906 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:12:56.158880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw"
Sep 16 12:13:41.744453 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.744375    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74b9d88877-srlcv"]
Sep 16 12:13:41.748044 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.748028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b9d88877-srlcv"
Sep 16 12:13:41.756237 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.756212    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74b9d88877-srlcv"]
Sep 16 12:13:41.814046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.814007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbf2m\" (UniqueName: \"kubernetes.io/projected/22a51a5f-252b-4927-bd48-67ac0c1cd524-kube-api-access-vbf2m\") pod \"authorino-74b9d88877-srlcv\" (UID: \"22a51a5f-252b-4927-bd48-67ac0c1cd524\") " pod="kuadrant-system/authorino-74b9d88877-srlcv"
Sep 16 12:13:41.814221 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.814127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22a51a5f-252b-4927-bd48-67ac0c1cd524-tls-cert\") pod \"authorino-74b9d88877-srlcv\" (UID: \"22a51a5f-252b-4927-bd48-67ac0c1cd524\") " pod="kuadrant-system/authorino-74b9d88877-srlcv"
Sep 16 12:13:41.915453 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.915415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22a51a5f-252b-4927-bd48-67ac0c1cd524-tls-cert\") pod \"authorino-74b9d88877-srlcv\" (UID: \"22a51a5f-252b-4927-bd48-67ac0c1cd524\") " pod="kuadrant-system/authorino-74b9d88877-srlcv"
Sep 16 12:13:41.915660 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.915480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbf2m\" (UniqueName: \"kubernetes.io/projected/22a51a5f-252b-4927-bd48-67ac0c1cd524-kube-api-access-vbf2m\") pod \"authorino-74b9d88877-srlcv\" (UID: \"22a51a5f-252b-4927-bd48-67ac0c1cd524\") " pod="kuadrant-system/authorino-74b9d88877-srlcv"
Sep 16 12:13:41.918030 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.918012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22a51a5f-252b-4927-bd48-67ac0c1cd524-tls-cert\") pod \"authorino-74b9d88877-srlcv\" (UID: \"22a51a5f-252b-4927-bd48-67ac0c1cd524\") " pod="kuadrant-system/authorino-74b9d88877-srlcv"
Sep 16 12:13:41.924440 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:41.924415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbf2m\" (UniqueName: \"kubernetes.io/projected/22a51a5f-252b-4927-bd48-67ac0c1cd524-kube-api-access-vbf2m\") pod \"authorino-74b9d88877-srlcv\" (UID: \"22a51a5f-252b-4927-bd48-67ac0c1cd524\") " pod="kuadrant-system/authorino-74b9d88877-srlcv"
Sep 16 12:13:42.058557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:42.058457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b9d88877-srlcv"
Sep 16 12:13:42.332091 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:42.332065    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74b9d88877-srlcv"]
Sep 16 12:13:42.333143 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:13:42.333119    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22a51a5f_252b_4927_bd48_67ac0c1cd524.slice/crio-723a1b5d4a37cdfca80212e821c70a200ad7b6225ee311f1c2713fca40dac37d WatchSource:0}: Error finding container 723a1b5d4a37cdfca80212e821c70a200ad7b6225ee311f1c2713fca40dac37d: Status 404 returned error can't find the container with id 723a1b5d4a37cdfca80212e821c70a200ad7b6225ee311f1c2713fca40dac37d
Sep 16 12:13:42.386021 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:42.385992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b9d88877-srlcv" event={"ID":"22a51a5f-252b-4927-bd48-67ac0c1cd524","Type":"ContainerStarted","Data":"723a1b5d4a37cdfca80212e821c70a200ad7b6225ee311f1c2713fca40dac37d"}
Sep 16 12:13:43.394467 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.394371    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b9d88877-srlcv" event={"ID":"22a51a5f-252b-4927-bd48-67ac0c1cd524","Type":"ContainerStarted","Data":"b5bdc9058a9ccc8a81c98271c106b10829e82a88cb350d8d9bd11eff25bdeb2e"}
Sep 16 12:13:43.421869 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.421813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74b9d88877-srlcv" podStartSLOduration=1.6393040129999998 podStartE2EDuration="2.421797777s" totalImagesPullingTime="782.493764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:13:43.420609713 +0000 UTC m=+713.057934553" watchObservedRunningTime="2026-09-16 12:13:43.421797777 +0000 UTC m=+713.059122660"
Sep 16 12:13:43.459298 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.459249    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f88d889d4-nr7hn"]
Sep 16 12:13:43.459820 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.459471    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-f88d889d4-nr7hn" podUID="36a598fc-5a9b-4daa-89e0-2c3e8b6b891c" containerName="authorino" containerID="cri-o://a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be" gracePeriod=30
Sep 16 12:13:43.703031 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.703006    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:13:43.830610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.830575    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-tls-cert\") pod \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\" (UID: \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\") "
Sep 16 12:13:43.830774 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.830632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rllxm\" (UniqueName: \"kubernetes.io/projected/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-kube-api-access-rllxm\") pod \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\" (UID: \"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c\") "
Sep 16 12:13:43.832860 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.832825    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-kube-api-access-rllxm" (OuterVolumeSpecName: "kube-api-access-rllxm") pod "36a598fc-5a9b-4daa-89e0-2c3e8b6b891c" (UID: "36a598fc-5a9b-4daa-89e0-2c3e8b6b891c"). InnerVolumeSpecName "kube-api-access-rllxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:43.841317 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.841290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "36a598fc-5a9b-4daa-89e0-2c3e8b6b891c" (UID: "36a598fc-5a9b-4daa-89e0-2c3e8b6b891c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:13:43.931531 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.931484    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-tls-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:13:43.931531 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:43.931523    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rllxm\" (UniqueName: \"kubernetes.io/projected/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c-kube-api-access-rllxm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:13:44.400622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.400528    2578 generic.go:356] "Generic (PLEG): container finished" podID="36a598fc-5a9b-4daa-89e0-2c3e8b6b891c" containerID="a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be" exitCode=0
Sep 16 12:13:44.400622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.400593    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f88d889d4-nr7hn"
Sep 16 12:13:44.401108 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.400609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f88d889d4-nr7hn" event={"ID":"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c","Type":"ContainerDied","Data":"a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be"}
Sep 16 12:13:44.401108 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.400658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f88d889d4-nr7hn" event={"ID":"36a598fc-5a9b-4daa-89e0-2c3e8b6b891c","Type":"ContainerDied","Data":"4547e8a3e6618a9bf615caa8e6f885716a2a9e55ba727518deaa13149b1425ee"}
Sep 16 12:13:44.401108 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.400682    2578 scope.go:117] "RemoveContainer" containerID="a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be"
Sep 16 12:13:44.411944 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.411905    2578 scope.go:117] "RemoveContainer" containerID="a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be"
Sep 16 12:13:44.412704 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:13:44.412638    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be\": container with ID starting with a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be not found: ID does not exist" containerID="a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be"
Sep 16 12:13:44.412704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.412681    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be"} err="failed to get container status \"a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be\": rpc error: code = NotFound desc = could not find container \"a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be\": container with ID starting with a4db2224b7c673d3c46d6afd0bacd58bc6461a4b4885830cb7ad6121ebf112be not found: ID does not exist"
Sep 16 12:13:44.436546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.436494    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f88d889d4-nr7hn"]
Sep 16 12:13:44.447359 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.447327    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f88d889d4-nr7hn"]
Sep 16 12:13:44.885706 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:13:44.885667    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a598fc-5a9b-4daa-89e0-2c3e8b6b891c" path="/var/lib/kubelet/pods/36a598fc-5a9b-4daa-89e0-2c3e8b6b891c/volumes"
Sep 16 12:14:08.051916 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.051883    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7490-9f8de8"]
Sep 16 12:14:08.052313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.052211    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a598fc-5a9b-4daa-89e0-2c3e8b6b891c" containerName="authorino"
Sep 16 12:14:08.052313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.052221    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a598fc-5a9b-4daa-89e0-2c3e8b6b891c" containerName="authorino"
Sep 16 12:14:08.052313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.052291    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a598fc-5a9b-4daa-89e0-2c3e8b6b891c" containerName="authorino"
Sep 16 12:14:08.056466 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.056449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-9f8de8"
Sep 16 12:14:08.059269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.059241    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zpzr9\""
Sep 16 12:14:08.061723 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.061703    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-9f8de8"]
Sep 16 12:14:08.148194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.148148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446g6\" (UniqueName: \"kubernetes.io/projected/215fbb5c-4473-4c6d-93df-da3885254e69-kube-api-access-446g6\") pod \"test-curl-7490-9f8de8\" (UID: \"215fbb5c-4473-4c6d-93df-da3885254e69\") " pod="opendatahub/test-curl-7490-9f8de8"
Sep 16 12:14:08.249609 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.249577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446g6\" (UniqueName: \"kubernetes.io/projected/215fbb5c-4473-4c6d-93df-da3885254e69-kube-api-access-446g6\") pod \"test-curl-7490-9f8de8\" (UID: \"215fbb5c-4473-4c6d-93df-da3885254e69\") " pod="opendatahub/test-curl-7490-9f8de8"
Sep 16 12:14:08.257847 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.257822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-446g6\" (UniqueName: \"kubernetes.io/projected/215fbb5c-4473-4c6d-93df-da3885254e69-kube-api-access-446g6\") pod \"test-curl-7490-9f8de8\" (UID: \"215fbb5c-4473-4c6d-93df-da3885254e69\") " pod="opendatahub/test-curl-7490-9f8de8"
Sep 16 12:14:08.366540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.366417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-9f8de8"
Sep 16 12:14:08.636283 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:08.636253    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-9f8de8"]
Sep 16 12:14:08.638173 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:08.638147    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod215fbb5c_4473_4c6d_93df_da3885254e69.slice/crio-dbed4fbe6aada8cb8d78f06271f7254380ab206cb3ff92aa64e241b3064ae831 WatchSource:0}: Error finding container dbed4fbe6aada8cb8d78f06271f7254380ab206cb3ff92aa64e241b3064ae831: Status 404 returned error can't find the container with id dbed4fbe6aada8cb8d78f06271f7254380ab206cb3ff92aa64e241b3064ae831
Sep 16 12:14:09.529592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:09.529550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-9f8de8" event={"ID":"215fbb5c-4473-4c6d-93df-da3885254e69","Type":"ContainerStarted","Data":"dbed4fbe6aada8cb8d78f06271f7254380ab206cb3ff92aa64e241b3064ae831"}
Sep 16 12:14:11.541709 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.541671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-9f8de8" event={"ID":"215fbb5c-4473-4c6d-93df-da3885254e69","Type":"ContainerStarted","Data":"c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf"}
Sep 16 12:14:11.563431 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.563382    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7490-9f8de8" podStartSLOduration=1.124191325 podStartE2EDuration="3.56336551s" totalImagesPullingTime="2.439174185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:11.562858654 +0000 UTC m=+741.200183492" watchObservedRunningTime="2026-09-16 12:14:11.56336551 +0000 UTC m=+741.200690349"
Sep 16 12:14:11.571472 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.571436    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"]
Sep 16 12:14:11.574851 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.574830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.577953 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.577928    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-641983-serving-cert\""
Sep 16 12:14:11.591482 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.591453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"]
Sep 16 12:14:11.682245 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.682198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a7cde5-d85c-4813-aa8c-1aae01f4e002-openshift-service-ca\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.682442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.682280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.682442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.682309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrvq\" (UniqueName: \"kubernetes.io/projected/58a7cde5-d85c-4813-aa8c-1aae01f4e002-kube-api-access-gbrvq\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.682442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.682352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-metrics-certs\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.783115 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.783077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-metrics-certs\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.783315 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.783135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a7cde5-d85c-4813-aa8c-1aae01f4e002-openshift-service-ca\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.783315 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.783193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.783315 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.783214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrvq\" (UniqueName: \"kubernetes.io/projected/58a7cde5-d85c-4813-aa8c-1aae01f4e002-kube-api-access-gbrvq\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.783488 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:11.783337    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-serving-cert: secret "maas-api-e2e-inf-a-641983-serving-cert" not found
Sep 16 12:14:11.783488 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:11.783410    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls podName:58a7cde5-d85c-4813-aa8c-1aae01f4e002 nodeName:}" failed. No retries permitted until 2026-09-16 12:14:12.28338851 +0000 UTC m=+741.920713329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls") pod "maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w" (UID: "58a7cde5-d85c-4813-aa8c-1aae01f4e002") : secret "maas-api-e2e-inf-a-641983-serving-cert" not found
Sep 16 12:14:11.783767 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.783748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a7cde5-d85c-4813-aa8c-1aae01f4e002-openshift-service-ca\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.785945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.785926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-metrics-certs\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:11.801000 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:11.800928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrvq\" (UniqueName: \"kubernetes.io/projected/58a7cde5-d85c-4813-aa8c-1aae01f4e002-kube-api-access-gbrvq\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:12.193525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.193413    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7490-9f8de8"]
Sep 16 12:14:12.205883 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.205851    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7490-9f8de8"]
Sep 16 12:14:12.288221 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.288183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:12.290799 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.290768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls\") pod \"maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:12.326497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.326464    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7490-eff52c"]
Sep 16 12:14:12.326819 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.326807    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215fbb5c-4473-4c6d-93df-da3885254e69" containerName="test-curl-7490-9f8de8"
Sep 16 12:14:12.326859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.326821    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="215fbb5c-4473-4c6d-93df-da3885254e69" containerName="test-curl-7490-9f8de8"
Sep 16 12:14:12.326905 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.326891    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="215fbb5c-4473-4c6d-93df-da3885254e69" containerName="test-curl-7490-9f8de8"
Sep 16 12:14:12.330057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.330039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-eff52c"
Sep 16 12:14:12.338639 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.338613    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-eff52c"]
Sep 16 12:14:12.486730 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.486685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:12.490886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.490866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2p5\" (UniqueName: \"kubernetes.io/projected/651755f6-35ee-48a1-965d-c4efe24c733c-kube-api-access-4x2p5\") pod \"test-curl-7490-eff52c\" (UID: \"651755f6-35ee-48a1-965d-c4efe24c733c\") " pod="opendatahub/test-curl-7490-eff52c"
Sep 16 12:14:12.548538 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:12.548488    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7490-9f8de8" container="test-curl-7490-9f8de8"
Sep 16 12:14:12.591897 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.591862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2p5\" (UniqueName: \"kubernetes.io/projected/651755f6-35ee-48a1-965d-c4efe24c733c-kube-api-access-4x2p5\") pod \"test-curl-7490-eff52c\" (UID: \"651755f6-35ee-48a1-965d-c4efe24c733c\") " pod="opendatahub/test-curl-7490-eff52c"
Sep 16 12:14:12.600196 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.600170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2p5\" (UniqueName: \"kubernetes.io/projected/651755f6-35ee-48a1-965d-c4efe24c733c-kube-api-access-4x2p5\") pod \"test-curl-7490-eff52c\" (UID: \"651755f6-35ee-48a1-965d-c4efe24c733c\") " pod="opendatahub/test-curl-7490-eff52c"
Sep 16 12:14:12.640662 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.640623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-eff52c"
Sep 16 12:14:12.650900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.650861    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"]
Sep 16 12:14:12.656679 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.656625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.659123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.659089    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"]
Sep 16 12:14:12.659503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.659465    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-556077-serving-cert\""
Sep 16 12:14:12.693314 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.693280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.693572 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.693341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0dc60-dd05-494b-b341-b97877d98ba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.693572 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.693423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-metrics-certs\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.693572 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.693444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vldw\" (UniqueName: \"kubernetes.io/projected/68d0dc60-dd05-494b-b341-b97877d98ba4-kube-api-access-7vldw\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.763999 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.763973    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"]
Sep 16 12:14:12.765174 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:12.765147    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a7cde5_d85c_4813_aa8c_1aae01f4e002.slice/crio-40b072cbee47e62b2ca803e54a687a2cf5c086cb157608823f7b396828a43364 WatchSource:0}: Error finding container 40b072cbee47e62b2ca803e54a687a2cf5c086cb157608823f7b396828a43364: Status 404 returned error can't find the container with id 40b072cbee47e62b2ca803e54a687a2cf5c086cb157608823f7b396828a43364
Sep 16 12:14:12.794648 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.794611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-metrics-certs\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.794648 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.794656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vldw\" (UniqueName: \"kubernetes.io/projected/68d0dc60-dd05-494b-b341-b97877d98ba4-kube-api-access-7vldw\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.794879 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.794707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.794879 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.794747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0dc60-dd05-494b-b341-b97877d98ba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.795496 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.795464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0dc60-dd05-494b-b341-b97877d98ba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.797405 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.797383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-metrics-certs\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.797499 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.797427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.808157 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.808132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vldw\" (UniqueName: \"kubernetes.io/projected/68d0dc60-dd05-494b-b341-b97877d98ba4-kube-api-access-7vldw\") pod \"maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:12.932680 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.932644    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-eff52c"]
Sep 16 12:14:12.936077 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:12.936050    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod651755f6_35ee_48a1_965d_c4efe24c733c.slice/crio-69ea6c6e59ebd3f837508659fe1cdbd689a202d9ccce7813644ffff73c1a3863 WatchSource:0}: Error finding container 69ea6c6e59ebd3f837508659fe1cdbd689a202d9ccce7813644ffff73c1a3863: Status 404 returned error can't find the container with id 69ea6c6e59ebd3f837508659fe1cdbd689a202d9ccce7813644ffff73c1a3863
Sep 16 12:14:12.977964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:12.977932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:13.274803 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.274768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"]
Sep 16 12:14:13.275892 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:13.275869    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d0dc60_dd05_494b_b341_b97877d98ba4.slice/crio-97c42eaeb3ebfbe66e11da13bbe051f427533bc5c8d32e354da7ca68e8e6bb7b WatchSource:0}: Error finding container 97c42eaeb3ebfbe66e11da13bbe051f427533bc5c8d32e354da7ca68e8e6bb7b: Status 404 returned error can't find the container with id 97c42eaeb3ebfbe66e11da13bbe051f427533bc5c8d32e354da7ca68e8e6bb7b
Sep 16 12:14:13.555334 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.555299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-eff52c" event={"ID":"651755f6-35ee-48a1-965d-c4efe24c733c","Type":"ContainerStarted","Data":"66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67"}
Sep 16 12:14:13.555864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.555347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-eff52c" event={"ID":"651755f6-35ee-48a1-965d-c4efe24c733c","Type":"ContainerStarted","Data":"69ea6c6e59ebd3f837508659fe1cdbd689a202d9ccce7813644ffff73c1a3863"}
Sep 16 12:14:13.557042 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.557017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w" event={"ID":"58a7cde5-d85c-4813-aa8c-1aae01f4e002","Type":"ContainerStarted","Data":"cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8"}
Sep 16 12:14:13.557148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.557075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w" event={"ID":"58a7cde5-d85c-4813-aa8c-1aae01f4e002","Type":"ContainerStarted","Data":"40b072cbee47e62b2ca803e54a687a2cf5c086cb157608823f7b396828a43364"}
Sep 16 12:14:13.557148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.557134    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:13.558285 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.558260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b" event={"ID":"68d0dc60-dd05-494b-b341-b97877d98ba4","Type":"ContainerStarted","Data":"97c42eaeb3ebfbe66e11da13bbe051f427533bc5c8d32e354da7ca68e8e6bb7b"}
Sep 16 12:14:13.558385 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.558350    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7490-9f8de8" podUID="215fbb5c-4473-4c6d-93df-da3885254e69" containerName="test-curl-7490-9f8de8" containerID="cri-o://c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf" gracePeriod=2
Sep 16 12:14:13.570678 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.570634    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7490-eff52c" podStartSLOduration=1.5706209260000001 podStartE2EDuration="1.570620926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:13.569044522 +0000 UTC m=+743.206369359" watchObservedRunningTime="2026-09-16 12:14:13.570620926 +0000 UTC m=+743.207945766"
Sep 16 12:14:13.593890 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.593840    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w" podStartSLOduration=1.954880593 podStartE2EDuration="2.59382621s" totalImagesPullingTime="638.945617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:13.59168373 +0000 UTC m=+743.229008601" watchObservedRunningTime="2026-09-16 12:14:13.59382621 +0000 UTC m=+743.231151048"
Sep 16 12:14:13.664182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.664144    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"]
Sep 16 12:14:13.669332 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.669309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.672726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.672702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ff057a-serving-cert\""
Sep 16 12:14:13.677576 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.677547    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"]
Sep 16 12:14:13.701388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.701349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87kpm\" (UniqueName: \"kubernetes.io/projected/69b5b387-09f6-415e-bba9-902c25a43bb9-kube-api-access-87kpm\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.701540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.701465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.701671 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.701550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b5b387-09f6-415e-bba9-902c25a43bb9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.701671 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.701633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-metrics-certs\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.802151 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.802112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.802151 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.802151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b5b387-09f6-415e-bba9-902c25a43bb9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.802404 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.802208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-metrics-certs\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.802404 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.802274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87kpm\" (UniqueName: \"kubernetes.io/projected/69b5b387-09f6-415e-bba9-902c25a43bb9-kube-api-access-87kpm\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.802404 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:13.802290    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-serving-cert: secret "maas-api-e2e-worker-w0-ff057a-serving-cert" not found
Sep 16 12:14:13.802404 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:13.802359    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls podName:69b5b387-09f6-415e-bba9-902c25a43bb9 nodeName:}" failed. No retries permitted until 2026-09-16 12:14:14.302335484 +0000 UTC m=+743.939660303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls") pod "maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh" (UID: "69b5b387-09f6-415e-bba9-902c25a43bb9") : secret "maas-api-e2e-worker-w0-ff057a-serving-cert" not found
Sep 16 12:14:13.803038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.803014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b5b387-09f6-415e-bba9-902c25a43bb9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.805173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.805154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-metrics-certs\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:13.813447 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:13.813420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87kpm\" (UniqueName: \"kubernetes.io/projected/69b5b387-09f6-415e-bba9-902c25a43bb9-kube-api-access-87kpm\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:14.227781 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:14.227690    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7490-eff52c"]
Sep 16 12:14:14.232084 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:14.232056    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7490-eff52c"]
Sep 16 12:14:14.305354 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:14.305310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:14.307859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:14.307839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:14.564870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:14.564777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b" event={"ID":"68d0dc60-dd05-494b-b341-b97877d98ba4","Type":"ContainerStarted","Data":"6188aca2029ea8ac94b7c21214fe0cdbea6c8be031d9683369084d4cc6b3b138"}
Sep 16 12:14:14.565305 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:14.565070    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7490-eff52c" container="test-curl-7490-eff52c"
Sep 16 12:14:14.585534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:14.585468    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b" podStartSLOduration=2.221928095 podStartE2EDuration="2.585451405s" totalImagesPullingTime="363.52331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:14.584551544 +0000 UTC m=+744.221876393" watchObservedRunningTime="2026-09-16 12:14:14.585451405 +0000 UTC m=+744.222776243"
Sep 16 12:14:14.601689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:14.601657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:14.873453 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:14.873423    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"]
Sep 16 12:14:14.874880 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:14.874850    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b5b387_09f6_415e_bba9_902c25a43bb9.slice/crio-d9f5d67c28aa4421c4b5023146fd0b1faf7d327e4f3e5957987e75238bbaf3d6 WatchSource:0}: Error finding container d9f5d67c28aa4421c4b5023146fd0b1faf7d327e4f3e5957987e75238bbaf3d6: Status 404 returned error can't find the container with id d9f5d67c28aa4421c4b5023146fd0b1faf7d327e4f3e5957987e75238bbaf3d6
Sep 16 12:14:15.572623 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.572581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh" event={"ID":"69b5b387-09f6-415e-bba9-902c25a43bb9","Type":"ContainerStarted","Data":"a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616"}
Sep 16 12:14:15.572989 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.572665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh" event={"ID":"69b5b387-09f6-415e-bba9-902c25a43bb9","Type":"ContainerStarted","Data":"d9f5d67c28aa4421c4b5023146fd0b1faf7d327e4f3e5957987e75238bbaf3d6"}
Sep 16 12:14:15.572989 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.572700    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7490-eff52c" podUID="651755f6-35ee-48a1-965d-c4efe24c733c" containerName="test-curl-7490-eff52c" containerID="cri-o://66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67" gracePeriod=2
Sep 16 12:14:15.572989 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.572830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:15.573241 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.573220    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:15.596913 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.596857    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh" podStartSLOduration=2.124983067 podStartE2EDuration="2.596833411s" totalImagesPullingTime="471.850344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:15.595756764 +0000 UTC m=+745.233081603" watchObservedRunningTime="2026-09-16 12:14:15.596833411 +0000 UTC m=+745.234158250"
Sep 16 12:14:15.739280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.739253    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-9f8de8"
Sep 16 12:14:15.818280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.818240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-446g6\" (UniqueName: \"kubernetes.io/projected/215fbb5c-4473-4c6d-93df-da3885254e69-kube-api-access-446g6\") pod \"215fbb5c-4473-4c6d-93df-da3885254e69\" (UID: \"215fbb5c-4473-4c6d-93df-da3885254e69\") "
Sep 16 12:14:15.820508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.820478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215fbb5c-4473-4c6d-93df-da3885254e69-kube-api-access-446g6" (OuterVolumeSpecName: "kube-api-access-446g6") pod "215fbb5c-4473-4c6d-93df-da3885254e69" (UID: "215fbb5c-4473-4c6d-93df-da3885254e69"). InnerVolumeSpecName "kube-api-access-446g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:14:15.919089 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:15.918997    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-446g6\" (UniqueName: \"kubernetes.io/projected/215fbb5c-4473-4c6d-93df-da3885254e69-kube-api-access-446g6\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:14:16.578774 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.578738    2578 generic.go:356] "Generic (PLEG): container finished" podID="215fbb5c-4473-4c6d-93df-da3885254e69" containerID="c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf" exitCode=137
Sep 16 12:14:16.579194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.578794    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-9f8de8"
Sep 16 12:14:16.579763 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.578832    2578 scope.go:117] "RemoveContainer" containerID="c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf"
Sep 16 12:14:16.591596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.591573    2578 scope.go:117] "RemoveContainer" containerID="c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf"
Sep 16 12:14:16.591942 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:16.591917    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf\": container with ID starting with c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf not found: ID does not exist" containerID="c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf"
Sep 16 12:14:16.592057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.591949    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf"} err="failed to get container status \"c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf\": rpc error: code = NotFound desc = could not find container \"c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf\": container with ID starting with c21576f7b122937f371640b1e4f574a66ec0ba2b472fdf15d2d7c807835ccecf not found: ID does not exist"
Sep 16 12:14:16.592313 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:16.592286    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7490-9f8de8\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" podUID="215fbb5c-4473-4c6d-93df-da3885254e69" pod="opendatahub/test-curl-7490-9f8de8"
Sep 16 12:14:16.884704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.884617    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215fbb5c-4473-4c6d-93df-da3885254e69" path="/var/lib/kubelet/pods/215fbb5c-4473-4c6d-93df-da3885254e69/volumes"
Sep 16 12:14:16.925170 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.925135    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"]
Sep 16 12:14:16.925497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.925485    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="651755f6-35ee-48a1-965d-c4efe24c733c" containerName="test-curl-7490-eff52c"
Sep 16 12:14:16.925559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.925499    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="651755f6-35ee-48a1-965d-c4efe24c733c" containerName="test-curl-7490-eff52c"
Sep 16 12:14:16.925611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.925601    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="651755f6-35ee-48a1-965d-c4efe24c733c" containerName="test-curl-7490-eff52c"
Sep 16 12:14:16.930186 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.930163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:16.933280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.933256    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-67e03001-openshift-default-dockercfg-cbzxb\""
Sep 16 12:14:16.933599 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.933581    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:14:16.945602 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:16.945567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"]
Sep 16 12:14:17.030155 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-socket\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.030335 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-podinfo\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.030335 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-token\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.030335 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smn5\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-kube-api-access-9smn5\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.030335 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-data\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.030475 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-credential-socket\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.030475 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-certs\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.030475 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-envoy\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.030475 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.030392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istiod-ca-cert\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131128 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smn5\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-kube-api-access-9smn5\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131338 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-data\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131338 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-credential-socket\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131338 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-certs\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131338 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-envoy\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131338 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istiod-ca-cert\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-socket\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-podinfo\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-token\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131750 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-credential-socket\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-certs\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-data\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.131898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.131855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-socket\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.132095 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.132072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istiod-ca-cert\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.133985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.133950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-envoy\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.134112 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.134019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-podinfo\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.140618 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.140556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smn5\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-kube-api-access-9smn5\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.140864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.140843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-token\") pod \"e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") " pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.240529 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.240475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:17.530118 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.530085    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"]
Sep 16 12:14:17.531467 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:17.531408    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bb16f2_82da_43f6_9e90_9e68bd9bccdc.slice/crio-6e47aa2555c10a09e2fc49ff120cbf81ed128034c4a4bc412734df8726968933 WatchSource:0}: Error finding container 6e47aa2555c10a09e2fc49ff120cbf81ed128034c4a4bc412734df8726968933: Status 404 returned error can't find the container with id 6e47aa2555c10a09e2fc49ff120cbf81ed128034c4a4bc412734df8726968933
Sep 16 12:14:17.586848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.586816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" event={"ID":"85bb16f2-82da-43f6-9e90-9e68bd9bccdc","Type":"ContainerStarted","Data":"6e47aa2555c10a09e2fc49ff120cbf81ed128034c4a4bc412734df8726968933"}
Sep 16 12:14:17.776782 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.776754    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-eff52c"
Sep 16 12:14:17.836747 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.836654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x2p5\" (UniqueName: \"kubernetes.io/projected/651755f6-35ee-48a1-965d-c4efe24c733c-kube-api-access-4x2p5\") pod \"651755f6-35ee-48a1-965d-c4efe24c733c\" (UID: \"651755f6-35ee-48a1-965d-c4efe24c733c\") "
Sep 16 12:14:17.838834 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.838810    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651755f6-35ee-48a1-965d-c4efe24c733c-kube-api-access-4x2p5" (OuterVolumeSpecName: "kube-api-access-4x2p5") pod "651755f6-35ee-48a1-965d-c4efe24c733c" (UID: "651755f6-35ee-48a1-965d-c4efe24c733c"). InnerVolumeSpecName "kube-api-access-4x2p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:14:17.937453 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:17.937414    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x2p5\" (UniqueName: \"kubernetes.io/projected/651755f6-35ee-48a1-965d-c4efe24c733c-kube-api-access-4x2p5\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:14:18.595932 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:18.595895    2578 generic.go:356] "Generic (PLEG): container finished" podID="651755f6-35ee-48a1-965d-c4efe24c733c" containerID="66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67" exitCode=137
Sep 16 12:14:18.596430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:18.595955    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-eff52c"
Sep 16 12:14:18.596430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:18.596061    2578 scope.go:117] "RemoveContainer" containerID="66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67"
Sep 16 12:14:18.609152 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:18.609114    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7490-eff52c\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" podUID="651755f6-35ee-48a1-965d-c4efe24c733c" pod="opendatahub/test-curl-7490-eff52c"
Sep 16 12:14:18.612230 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:18.612208    2578 scope.go:117] "RemoveContainer" containerID="66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67"
Sep 16 12:14:18.612677 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:18.612650    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67\": container with ID starting with 66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67 not found: ID does not exist" containerID="66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67"
Sep 16 12:14:18.612778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:18.612687    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67"} err="failed to get container status \"66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67\": rpc error: code = NotFound desc = could not find container \"66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67\": container with ID starting with 66d1ce1486bfad5ff56bdf9af292973023fdc64a887bd62709cb6896bcb63f67 not found: ID does not exist"
Sep 16 12:14:18.916077 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:18.915981    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="651755f6-35ee-48a1-965d-c4efe24c733c" path="/var/lib/kubelet/pods/651755f6-35ee-48a1-965d-c4efe24c733c/volumes"
Sep 16 12:14:19.571131 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:19.571087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:14:20.476943 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:20.476913    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:14:20.477203 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:20.476984    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:14:20.477203 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:20.477012    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:14:20.610397 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:20.610361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" event={"ID":"85bb16f2-82da-43f6-9e90-9e68bd9bccdc","Type":"ContainerStarted","Data":"afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993"}
Sep 16 12:14:20.640305 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:20.640236    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" podStartSLOduration=1.697785659 podStartE2EDuration="4.64021752s" totalImagesPullingTime="2.942431861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:20.639058476 +0000 UTC m=+750.276383315" watchObservedRunningTime="2026-09-16 12:14:20.64021752 +0000 UTC m=+750.277542393"
Sep 16 12:14:21.240831 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:21.240787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:21.246037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:21.246006    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:21.585000 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:21.584916    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:14:21.615864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:21.615832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:21.617005 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:21.616982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:22.593951 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:22.593924    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:14:26.275826 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.275795    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"]
Sep 16 12:14:26.279661 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.279640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.282108 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.282084    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-67e03001-serving-cert\""
Sep 16 12:14:26.294701 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.294677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"]
Sep 16 12:14:26.318219 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.318190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11551590-6a51-4f9f-b678-94a5dcf2718a-openshift-service-ca\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.318219 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.318220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.318360 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.318238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-metrics-certs\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.318360 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.318257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqfs\" (UniqueName: \"kubernetes.io/projected/11551590-6a51-4f9f-b678-94a5dcf2718a-kube-api-access-cxqfs\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.419465 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.419429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11551590-6a51-4f9f-b678-94a5dcf2718a-openshift-service-ca\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.419465 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.419467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.419696 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.419494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-metrics-certs\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.419696 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.419536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqfs\" (UniqueName: \"kubernetes.io/projected/11551590-6a51-4f9f-b678-94a5dcf2718a-kube-api-access-cxqfs\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.419696 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:26.419640    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-serving-cert: secret "maas-api-e2e-ait-67e03001-serving-cert" not found
Sep 16 12:14:26.419818 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:26.419725    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls podName:11551590-6a51-4f9f-b678-94a5dcf2718a nodeName:}" failed. No retries permitted until 2026-09-16 12:14:26.919703764 +0000 UTC m=+756.557028582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls") pod "maas-api-e2e-ait-67e03001-595b5f9666-6ngd6" (UID: "11551590-6a51-4f9f-b678-94a5dcf2718a") : secret "maas-api-e2e-ait-67e03001-serving-cert" not found
Sep 16 12:14:26.420168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.420139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11551590-6a51-4f9f-b678-94a5dcf2718a-openshift-service-ca\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.422134 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.422115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-metrics-certs\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.435213 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.435188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqfs\" (UniqueName: \"kubernetes.io/projected/11551590-6a51-4f9f-b678-94a5dcf2718a-kube-api-access-cxqfs\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.925368 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.925337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:26.927780 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:26.927756    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls\") pod \"maas-api-e2e-ait-67e03001-595b5f9666-6ngd6\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:27.190401 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.190312    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:27.465337 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.465307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"]
Sep 16 12:14:27.466882 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:27.466857    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11551590_6a51_4f9f_b678_94a5dcf2718a.slice/crio-23b85c6fb10603fb5a2326a0bb6c6295284f2232ab32ec1c27b58a021485eafb WatchSource:0}: Error finding container 23b85c6fb10603fb5a2326a0bb6c6295284f2232ab32ec1c27b58a021485eafb: Status 404 returned error can't find the container with id 23b85c6fb10603fb5a2326a0bb6c6295284f2232ab32ec1c27b58a021485eafb
Sep 16 12:14:27.650498 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.650463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6" event={"ID":"11551590-6a51-4f9f-b678-94a5dcf2718a","Type":"ContainerStarted","Data":"23b85c6fb10603fb5a2326a0bb6c6295284f2232ab32ec1c27b58a021485eafb"}
Sep 16 12:14:27.857651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.857568    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"]
Sep 16 12:14:27.862862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.862840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:27.865522 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.865482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-dc3e2c-serving-cert\""
Sep 16 12:14:27.873012 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.872993    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"]
Sep 16 12:14:27.934057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.934027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr7pg\" (UniqueName: \"kubernetes.io/projected/16d0498f-5e83-4ac7-b34c-d33928938b15-kube-api-access-dr7pg\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:27.934245 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.934066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d0498f-5e83-4ac7-b34c-d33928938b15-openshift-service-ca\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:27.934245 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.934207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-metrics-certs\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:27.934245 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:27.934242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.035355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.035318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d0498f-5e83-4ac7-b34c-d33928938b15-openshift-service-ca\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.035546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.035444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-metrics-certs\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.035546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.035484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.035546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.035542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr7pg\" (UniqueName: \"kubernetes.io/projected/16d0498f-5e83-4ac7-b34c-d33928938b15-kube-api-access-dr7pg\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.035737 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:28.035630    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-serving-cert: secret "maas-api-e2e-autores-dc3e2c-serving-cert" not found
Sep 16 12:14:28.035737 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:28.035706    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls podName:16d0498f-5e83-4ac7-b34c-d33928938b15 nodeName:}" failed. No retries permitted until 2026-09-16 12:14:28.535684428 +0000 UTC m=+758.173009245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls") pod "maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn" (UID: "16d0498f-5e83-4ac7-b34c-d33928938b15") : secret "maas-api-e2e-autores-dc3e2c-serving-cert" not found
Sep 16 12:14:28.036056 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.036037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d0498f-5e83-4ac7-b34c-d33928938b15-openshift-service-ca\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.038201 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.038181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-metrics-certs\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.044787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.044764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr7pg\" (UniqueName: \"kubernetes.io/projected/16d0498f-5e83-4ac7-b34c-d33928938b15-kube-api-access-dr7pg\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.540897 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.540854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.543417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.543395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls\") pod \"maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:28.657193 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.657157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6" event={"ID":"11551590-6a51-4f9f-b678-94a5dcf2718a","Type":"ContainerStarted","Data":"10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917"}
Sep 16 12:14:28.657381 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.657362    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:28.675669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.675622    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6" podStartSLOduration=2.104886967 podStartE2EDuration="2.6755992s" totalImagesPullingTime="570.712233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:28.674018951 +0000 UTC m=+758.311343792" watchObservedRunningTime="2026-09-16 12:14:28.6755992 +0000 UTC m=+758.312924037"
Sep 16 12:14:28.776341 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:28.776304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:29.050852 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:29.050824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"]
Sep 16 12:14:29.051617 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:29.051590    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d0498f_5e83_4ac7_b34c_d33928938b15.slice/crio-b04eaadbf2cb38ad06cb5d7acf2773f4569786077a012140939de4ca2b9563ae WatchSource:0}: Error finding container b04eaadbf2cb38ad06cb5d7acf2773f4569786077a012140939de4ca2b9563ae: Status 404 returned error can't find the container with id b04eaadbf2cb38ad06cb5d7acf2773f4569786077a012140939de4ca2b9563ae
Sep 16 12:14:29.670124 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:29.670091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn" event={"ID":"16d0498f-5e83-4ac7-b34c-d33928938b15","Type":"ContainerStarted","Data":"f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2"}
Sep 16 12:14:29.670609 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:29.670169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn" event={"ID":"16d0498f-5e83-4ac7-b34c-d33928938b15","Type":"ContainerStarted","Data":"b04eaadbf2cb38ad06cb5d7acf2773f4569786077a012140939de4ca2b9563ae"}
Sep 16 12:14:29.689705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:29.689619    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn" podStartSLOduration=2.32042541 podStartE2EDuration="2.689605443s" totalImagesPullingTime="369.180033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:29.687653874 +0000 UTC m=+759.324978712" watchObservedRunningTime="2026-09-16 12:14:29.689605443 +0000 UTC m=+759.326930282"
Sep 16 12:14:30.675154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:30.675068    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:33.479049 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.479007    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"]
Sep 16 12:14:33.483249 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.483219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.486618 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.486594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-08170a-serving-cert\""
Sep 16 12:14:33.494492 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.494459    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"]
Sep 16 12:14:33.588150 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.588115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-metrics-certs\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.588321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.588180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.588321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.588202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.588321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.588227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6czw\" (UniqueName: \"kubernetes.io/projected/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-kube-api-access-f6czw\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.688735 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.688694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.688735 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.688727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.689046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.688760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6czw\" (UniqueName: \"kubernetes.io/projected/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-kube-api-access-f6czw\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.689046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.688814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-metrics-certs\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.689046 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:33.688929    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-serving-cert: secret "maas-api-e2e-inf-b-08170a-serving-cert" not found
Sep 16 12:14:33.689046 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:14:33.689025    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls podName:659f3f12-e436-4b61-a5fd-9b36cfa7db7a nodeName:}" failed. No retries permitted until 2026-09-16 12:14:34.189000309 +0000 UTC m=+763.826325131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls") pod "maas-api-e2e-inf-b-08170a-5877557b75-82qrl" (UID: "659f3f12-e436-4b61-a5fd-9b36cfa7db7a") : secret "maas-api-e2e-inf-b-08170a-serving-cert" not found
Sep 16 12:14:33.689506 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.689474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.691493 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.691464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-metrics-certs\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:33.696910 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:33.696888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6czw\" (UniqueName: \"kubernetes.io/projected/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-kube-api-access-f6czw\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:34.194046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:34.194009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:34.196478 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:34.196448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls\") pod \"maas-api-e2e-inf-b-08170a-5877557b75-82qrl\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:34.398094 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:34.398059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:34.670486 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:34.670453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"]
Sep 16 12:14:34.671837 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:14:34.671811    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659f3f12_e436_4b61_a5fd_9b36cfa7db7a.slice/crio-7c69eaebd2e32389b2a64f8cfd431faf4b1240344c2dadd4eca0175fc0c7c961 WatchSource:0}: Error finding container 7c69eaebd2e32389b2a64f8cfd431faf4b1240344c2dadd4eca0175fc0c7c961: Status 404 returned error can't find the container with id 7c69eaebd2e32389b2a64f8cfd431faf4b1240344c2dadd4eca0175fc0c7c961
Sep 16 12:14:34.675768 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:34.675748    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:14:34.699567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:34.699488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl" event={"ID":"659f3f12-e436-4b61-a5fd-9b36cfa7db7a","Type":"ContainerStarted","Data":"7c69eaebd2e32389b2a64f8cfd431faf4b1240344c2dadd4eca0175fc0c7c961"}
Sep 16 12:14:35.706701 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:35.706668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl" event={"ID":"659f3f12-e436-4b61-a5fd-9b36cfa7db7a","Type":"ContainerStarted","Data":"6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f"}
Sep 16 12:14:35.707092 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:35.706782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:35.723806 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:35.723742    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl" podStartSLOduration=2.222852722 podStartE2EDuration="2.723727723s" totalImagesPullingTime="500.875001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:35.722059083 +0000 UTC m=+765.359383921" watchObservedRunningTime="2026-09-16 12:14:35.723727723 +0000 UTC m=+765.361052561"
Sep 16 12:14:36.685798 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:36.685771    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:14:41.717802 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:41.717768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:14:43.750531 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:43.750431    2578 generic.go:356] "Generic (PLEG): container finished" podID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerID="afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993" exitCode=0
Sep 16 12:14:43.750531 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:43.750503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" event={"ID":"85bb16f2-82da-43f6-9e90-9e68bd9bccdc","Type":"ContainerDied","Data":"afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993"}
Sep 16 12:14:43.750963 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:43.750898    2578 scope.go:117] "RemoveContainer" containerID="afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993"
Sep 16 12:14:43.751976 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:43.751949    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:14:43.752033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:43.752016    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:14:43.752069 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:43.752047    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:14:44.760041 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:44.760008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" event={"ID":"85bb16f2-82da-43f6-9e90-9e68bd9bccdc","Type":"ContainerStarted","Data":"b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9"}
Sep 16 12:14:45.241822 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:45.241735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:45.242467 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:45.242131    2578 patch_prober.go:28] interesting pod/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:14:45.242467 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:45.242198    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 12:14:46.240973 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:46.240932    2578 patch_prober.go:28] interesting pod/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:14:46.241588 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:46.241554    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 12:14:47.241443 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:47.241404    2578 patch_prober.go:28] interesting pod/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:14:47.242077 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:47.242041    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 12:14:48.241017 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:48.240975    2578 patch_prober.go:28] interesting pod/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:14:48.241322 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:48.241288    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 12:14:49.240913 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:49.240867    2578 patch_prober.go:28] interesting pod/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:14:49.241422 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:49.240936    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 12:14:50.296858 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:50.296822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:50.297312 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:50.297223    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:14:50.298052 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:14:50.298028    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:15:04.228396 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.228361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"]
Sep 16 12:15:04.228871 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.228659    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6" podUID="11551590-6a51-4f9f-b678-94a5dcf2718a" containerName="maas-api" containerID="cri-o://10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917" gracePeriod=30
Sep 16 12:15:04.483578 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.483496    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:15:04.558446 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.558399    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"]
Sep 16 12:15:04.558959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.558905    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy" containerID="cri-o://b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9" gracePeriod=30
Sep 16 12:15:04.606263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.606229    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11551590-6a51-4f9f-b678-94a5dcf2718a-openshift-service-ca\") pod \"11551590-6a51-4f9f-b678-94a5dcf2718a\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") "
Sep 16 12:15:04.606428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.606315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls\") pod \"11551590-6a51-4f9f-b678-94a5dcf2718a\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") "
Sep 16 12:15:04.606503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.606456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxqfs\" (UniqueName: \"kubernetes.io/projected/11551590-6a51-4f9f-b678-94a5dcf2718a-kube-api-access-cxqfs\") pod \"11551590-6a51-4f9f-b678-94a5dcf2718a\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") "
Sep 16 12:15:04.606585 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.606561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-metrics-certs\") pod \"11551590-6a51-4f9f-b678-94a5dcf2718a\" (UID: \"11551590-6a51-4f9f-b678-94a5dcf2718a\") "
Sep 16 12:15:04.606874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.606843    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11551590-6a51-4f9f-b678-94a5dcf2718a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11551590-6a51-4f9f-b678-94a5dcf2718a" (UID: "11551590-6a51-4f9f-b678-94a5dcf2718a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:04.608818 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.608789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "11551590-6a51-4f9f-b678-94a5dcf2718a" (UID: "11551590-6a51-4f9f-b678-94a5dcf2718a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:15:04.608922 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.608811    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11551590-6a51-4f9f-b678-94a5dcf2718a" (UID: "11551590-6a51-4f9f-b678-94a5dcf2718a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:15:04.609123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.609102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11551590-6a51-4f9f-b678-94a5dcf2718a-kube-api-access-cxqfs" (OuterVolumeSpecName: "kube-api-access-cxqfs") pod "11551590-6a51-4f9f-b678-94a5dcf2718a" (UID: "11551590-6a51-4f9f-b678-94a5dcf2718a"). InnerVolumeSpecName "kube-api-access-cxqfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:04.708038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.708009    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:04.708038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.708036    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxqfs\" (UniqueName: \"kubernetes.io/projected/11551590-6a51-4f9f-b678-94a5dcf2718a-kube-api-access-cxqfs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:04.708240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.708047    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11551590-6a51-4f9f-b678-94a5dcf2718a-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:04.708240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.708056    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11551590-6a51-4f9f-b678-94a5dcf2718a-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:04.879335 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.879255    2578 generic.go:356] "Generic (PLEG): container finished" podID="11551590-6a51-4f9f-b678-94a5dcf2718a" containerID="10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917" exitCode=0
Sep 16 12:15:04.879480 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.879362    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"
Sep 16 12:15:04.884053 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.884028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6" event={"ID":"11551590-6a51-4f9f-b678-94a5dcf2718a","Type":"ContainerDied","Data":"10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917"}
Sep 16 12:15:04.884182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.884071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6" event={"ID":"11551590-6a51-4f9f-b678-94a5dcf2718a","Type":"ContainerDied","Data":"23b85c6fb10603fb5a2326a0bb6c6295284f2232ab32ec1c27b58a021485eafb"}
Sep 16 12:15:04.884182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.884104    2578 scope.go:117] "RemoveContainer" containerID="10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917"
Sep 16 12:15:04.894851 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.894806    2578 scope.go:117] "RemoveContainer" containerID="10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917"
Sep 16 12:15:04.895086 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:15:04.895066    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917\": container with ID starting with 10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917 not found: ID does not exist" containerID="10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917"
Sep 16 12:15:04.895150 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.895096    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917"} err="failed to get container status \"10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917\": rpc error: code = NotFound desc = could not find container \"10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917\": container with ID starting with 10b6a222ecdb1a6a62f9e2127d199b5392c8c483ecd708e3eecdbb74e9342917 not found: ID does not exist"
Sep 16 12:15:04.903687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.903660    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"]
Sep 16 12:15:04.907873 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:04.907852    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67e03001-595b5f9666-6ngd6"]
Sep 16 12:15:05.300398 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:05.300363    2578 patch_prober.go:28] interesting pod/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:15:05.300794 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:05.300416    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:15:06.885107 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:06.885072    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11551590-6a51-4f9f-b678-94a5dcf2718a" path="/var/lib/kubelet/pods/11551590-6a51-4f9f-b678-94a5dcf2718a/volumes"
Sep 16 12:15:09.391033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.390994    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"]
Sep 16 12:15:09.391644 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.391625    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11551590-6a51-4f9f-b678-94a5dcf2718a" containerName="maas-api"
Sep 16 12:15:09.391703 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.391648    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="11551590-6a51-4f9f-b678-94a5dcf2718a" containerName="maas-api"
Sep 16 12:15:09.391789 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.391777    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="11551590-6a51-4f9f-b678-94a5dcf2718a" containerName="maas-api"
Sep 16 12:15:09.396530 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.396498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.399017 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.398997    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f74dc7e1-serving-cert\""
Sep 16 12:15:09.403587 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.403561    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"]
Sep 16 12:15:09.452181 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.452144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.452363 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.452203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893ad837-39fd-4cab-b72a-f1a4dd7129e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.452363 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.452254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-metrics-certs\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.452363 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.452287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zbb\" (UniqueName: \"kubernetes.io/projected/893ad837-39fd-4cab-b72a-f1a4dd7129e9-kube-api-access-z7zbb\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.553092 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.553037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.553303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.553124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893ad837-39fd-4cab-b72a-f1a4dd7129e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.553303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.553163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-metrics-certs\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.553303 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:15:09.553192    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-serving-cert: secret "maas-api-e2e-migrate-f74dc7e1-serving-cert" not found
Sep 16 12:15:09.553303 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:15:09.553286    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls podName:893ad837-39fd-4cab-b72a-f1a4dd7129e9 nodeName:}" failed. No retries permitted until 2026-09-16 12:15:10.053261226 +0000 UTC m=+799.690586042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls") pod "maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn" (UID: "893ad837-39fd-4cab-b72a-f1a4dd7129e9") : secret "maas-api-e2e-migrate-f74dc7e1-serving-cert" not found
Sep 16 12:15:09.553303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.553306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zbb\" (UniqueName: \"kubernetes.io/projected/893ad837-39fd-4cab-b72a-f1a4dd7129e9-kube-api-access-z7zbb\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.553941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.553922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893ad837-39fd-4cab-b72a-f1a4dd7129e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.555805 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.555782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-metrics-certs\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.563497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.563476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zbb\" (UniqueName: \"kubernetes.io/projected/893ad837-39fd-4cab-b72a-f1a4dd7129e9-kube-api-access-z7zbb\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:09.817251 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.817225    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:15:09.907818 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.907786    2578 generic.go:356] "Generic (PLEG): container finished" podID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerID="b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9" exitCode=0
Sep 16 12:15:09.907991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.907827    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"
Sep 16 12:15:09.907991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.907860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" event={"ID":"85bb16f2-82da-43f6-9e90-9e68bd9bccdc","Type":"ContainerDied","Data":"b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9"}
Sep 16 12:15:09.907991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.907920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc" event={"ID":"85bb16f2-82da-43f6-9e90-9e68bd9bccdc","Type":"ContainerDied","Data":"6e47aa2555c10a09e2fc49ff120cbf81ed128034c4a4bc412734df8726968933"}
Sep 16 12:15:09.907991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.907942    2578 scope.go:117] "RemoveContainer" containerID="b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9"
Sep 16 12:15:09.917620 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.917600    2578 scope.go:117] "RemoveContainer" containerID="afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993"
Sep 16 12:15:09.928392 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.928372    2578 scope.go:117] "RemoveContainer" containerID="b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9"
Sep 16 12:15:09.928793 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:15:09.928773    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9\": container with ID starting with b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9 not found: ID does not exist" containerID="b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9"
Sep 16 12:15:09.928867 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.928805    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9"} err="failed to get container status \"b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9\": rpc error: code = NotFound desc = could not find container \"b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9\": container with ID starting with b8e1ec63f53f549d3c6172be33ea2cfdf26f45aad9cb14de5721d4edb79ce7f9 not found: ID does not exist"
Sep 16 12:15:09.928867 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.928824    2578 scope.go:117] "RemoveContainer" containerID="afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993"
Sep 16 12:15:09.929118 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:15:09.929100    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993\": container with ID starting with afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993 not found: ID does not exist" containerID="afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993"
Sep 16 12:15:09.929179 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.929122    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993"} err="failed to get container status \"afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993\": rpc error: code = NotFound desc = could not find container \"afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993\": container with ID starting with afb496be442b5567c178bc9cc3ef3162085b06828177852afc963c413b10b993 not found: ID does not exist"
Sep 16 12:15:09.956395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-certs\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956488 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956440    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-credential-socket\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956488 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956484    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9smn5\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-kube-api-access-9smn5\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956593 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956536    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-data\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956593 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956564    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-envoy\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956591    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istiod-ca-cert\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-podinfo\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956794 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-token\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956794 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-socket\") pod \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\" (UID: \"85bb16f2-82da-43f6-9e90-9e68bd9bccdc\") "
Sep 16 12:15:09.956794 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:09.956794 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956749    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:09.957009 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-data" (OuterVolumeSpecName: "istio-data") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:09.957009 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.956970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:09.957128 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.957073    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-credential-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:09.957128 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.957094    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-data\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:09.957128 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.957098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:09.957128 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.957107    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istiod-ca-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:09.957275 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.957141    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:09.958913 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.958890    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:15:09.959005 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.958984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-token" (OuterVolumeSpecName: "istio-token") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:09.959274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.959252    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:09.959338 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:09.959269    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-kube-api-access-9smn5" (OuterVolumeSpecName: "kube-api-access-9smn5") pod "85bb16f2-82da-43f6-9e90-9e68bd9bccdc" (UID: "85bb16f2-82da-43f6-9e90-9e68bd9bccdc"). InnerVolumeSpecName "kube-api-access-9smn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:10.058341 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.058301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:10.058602 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.058506    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-podinfo\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:10.058689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.058610    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-token\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:10.058689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.058624    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-workload-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:10.058689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.058639    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smn5\" (UniqueName: \"kubernetes.io/projected/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-kube-api-access-9smn5\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:10.058689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.058654    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bb16f2-82da-43f6-9e90-9e68bd9bccdc-istio-envoy\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:10.060730 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.060704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls\") pod \"maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:10.233074 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.233039    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"]
Sep 16 12:15:10.236954 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.236929    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-67e03001-openshift-default-74cfcf755b-8nvdc"]
Sep 16 12:15:10.309407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.309368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:10.333939 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.333909    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"]
Sep 16 12:15:10.334336 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.334320    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy"
Sep 16 12:15:10.334417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.334340    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy"
Sep 16 12:15:10.334480 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.334467    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy"
Sep 16 12:15:10.334552 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.334482    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy"
Sep 16 12:15:10.337677 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.337656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.340146 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.340124    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"openshift-service-ca.crt\""
Sep 16 12:15:10.340146 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.340135    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"default-dockercfg-46bj7\""
Sep 16 12:15:10.340317 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.340271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"kube-root-ca.crt\""
Sep 16 12:15:10.340363 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.340320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"simulator-w4-556077-kserve-self-signed-certs\""
Sep 16 12:15:10.347719 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.347699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"]
Sep 16 12:15:10.462773 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.462743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-dshm\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.463179 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.462811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1302f9e5-1121-47d1-9a5b-821fcc994388-tls-certs\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.463179 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.462841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-tmp-dir\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.463179 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.462868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfrj\" (UniqueName: \"kubernetes.io/projected/1302f9e5-1121-47d1-9a5b-821fcc994388-kube-api-access-nvfrj\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.463179 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.462896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-model-cache\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.463179 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.462951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-home\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.564243 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.564199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-home\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.564375 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.564252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-dshm\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.564375 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.564324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1302f9e5-1121-47d1-9a5b-821fcc994388-tls-certs\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.564375 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.564355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-tmp-dir\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.564671 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.564393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfrj\" (UniqueName: \"kubernetes.io/projected/1302f9e5-1121-47d1-9a5b-821fcc994388-kube-api-access-nvfrj\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.564671 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.564419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-model-cache\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.564779 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.564750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-home\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.564833 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.564808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-model-cache\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.565236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.565071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-tmp-dir\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.567429 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.567407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-dshm\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.567653 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.567620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1302f9e5-1121-47d1-9a5b-821fcc994388-tls-certs\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.573873 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.573848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfrj\" (UniqueName: \"kubernetes.io/projected/1302f9e5-1121-47d1-9a5b-821fcc994388-kube-api-access-nvfrj\") pod \"simulator-w4-556077-kserve-54d58c7d75-vqxqn\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") " pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.587376 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.587353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"]
Sep 16 12:15:10.588200 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:15:10.588182    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893ad837_39fd_4cab_b72a_f1a4dd7129e9.slice/crio-5f16e1db152098bd12e9a586c74082cc963f124eb1e32a43da151d5921af9e8b WatchSource:0}: Error finding container 5f16e1db152098bd12e9a586c74082cc963f124eb1e32a43da151d5921af9e8b: Status 404 returned error can't find the container with id 5f16e1db152098bd12e9a586c74082cc963f124eb1e32a43da151d5921af9e8b
Sep 16 12:15:10.647663 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.647628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:10.884572 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.884445    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" path="/var/lib/kubelet/pods/85bb16f2-82da-43f6-9e90-9e68bd9bccdc/volumes"
Sep 16 12:15:10.914359 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.914327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn" event={"ID":"893ad837-39fd-4cab-b72a-f1a4dd7129e9","Type":"ContainerStarted","Data":"5f16e1db152098bd12e9a586c74082cc963f124eb1e32a43da151d5921af9e8b"}
Sep 16 12:15:10.919955 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:10.919932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"]
Sep 16 12:15:10.921301 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:15:10.921275    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1302f9e5_1121_47d1_9a5b_821fcc994388.slice/crio-e8799c14bfc37abec18c2fe9b76ec945080065b535eae492e92c23a57ab7ccf2 WatchSource:0}: Error finding container e8799c14bfc37abec18c2fe9b76ec945080065b535eae492e92c23a57ab7ccf2: Status 404 returned error can't find the container with id e8799c14bfc37abec18c2fe9b76ec945080065b535eae492e92c23a57ab7ccf2
Sep 16 12:15:11.922096 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:11.922001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn" event={"ID":"893ad837-39fd-4cab-b72a-f1a4dd7129e9","Type":"ContainerStarted","Data":"d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34"}
Sep 16 12:15:11.922096 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:11.922067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:11.923845 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:11.923821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn" event={"ID":"1302f9e5-1121-47d1-9a5b-821fcc994388","Type":"ContainerStarted","Data":"ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d"}
Sep 16 12:15:11.923965 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:11.923855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn" event={"ID":"1302f9e5-1121-47d1-9a5b-821fcc994388","Type":"ContainerStarted","Data":"e8799c14bfc37abec18c2fe9b76ec945080065b535eae492e92c23a57ab7ccf2"}
Sep 16 12:15:11.939943 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:11.939896    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn" podStartSLOduration=2.328943108 podStartE2EDuration="2.939878681s" totalImagesPullingTime="610.935573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:11.939219781 +0000 UTC m=+801.576544619" watchObservedRunningTime="2026-09-16 12:15:11.939878681 +0000 UTC m=+801.577203519"
Sep 16 12:15:11.957220 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:11.957179    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn" podStartSLOduration=1.957166674 podStartE2EDuration="1.957166674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:11.955404996 +0000 UTC m=+801.592729834" watchObservedRunningTime="2026-09-16 12:15:11.957166674 +0000 UTC m=+801.594491564"
Sep 16 12:15:17.935018 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:17.934980    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:20.648273 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:20.648238    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:20.648710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:20.648295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:20.661061 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:20.661024    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:20.981962 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:20.981933    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:15:26.037923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.037891    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"]
Sep 16 12:15:26.038448 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.038429    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy"
Sep 16 12:15:26.038551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.038451    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bb16f2-82da-43f6-9e90-9e68bd9bccdc" containerName="istio-proxy"
Sep 16 12:15:26.041698 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.041678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.045558 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.045454    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"kube-root-ca.crt\""
Sep 16 12:15:26.045558 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.045478    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"openshift-service-ca.crt\""
Sep 16 12:15:26.045769 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.045476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"default-dockercfg-szmz8\""
Sep 16 12:15:26.045769 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.045564    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"simulator-w0-ff057a-kserve-self-signed-certs\""
Sep 16 12:15:26.049175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.049153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"]
Sep 16 12:15:26.206323 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.206285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpr8t\" (UniqueName: \"kubernetes.io/projected/03188b71-c35e-48cc-a837-6d024c966f4b-kube-api-access-dpr8t\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.206534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.206350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-tmp-dir\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.206534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.206427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188b71-c35e-48cc-a837-6d024c966f4b-tls-certs\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.206534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.206461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-dshm\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.206534 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.206496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-home\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.206705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.206546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-model-cache\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.307571 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.307453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-dshm\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.307571 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.307537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-home\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.307571 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.307578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-model-cache\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.307843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.307641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpr8t\" (UniqueName: \"kubernetes.io/projected/03188b71-c35e-48cc-a837-6d024c966f4b-kube-api-access-dpr8t\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.307843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.307709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-tmp-dir\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.307843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.307766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188b71-c35e-48cc-a837-6d024c966f4b-tls-certs\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.308033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.308010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-home\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.308086 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.308042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-model-cache\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.308126 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.308082    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-tmp-dir\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.309928 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.309901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-dshm\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.310194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.310178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188b71-c35e-48cc-a837-6d024c966f4b-tls-certs\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.316459 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.316441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpr8t\" (UniqueName: \"kubernetes.io/projected/03188b71-c35e-48cc-a837-6d024c966f4b-kube-api-access-dpr8t\") pod \"simulator-w0-ff057a-kserve-6f899fb98d-znrjp\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") " pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.352496 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.352469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:26.624479 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:26.624453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"]
Sep 16 12:15:26.625612 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:15:26.625583    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03188b71_c35e_48cc_a837_6d024c966f4b.slice/crio-ba185de0063f5ebf17a1782e74c157775d1d5a68d4d48be3828a2320f3ac941b WatchSource:0}: Error finding container ba185de0063f5ebf17a1782e74c157775d1d5a68d4d48be3828a2320f3ac941b: Status 404 returned error can't find the container with id ba185de0063f5ebf17a1782e74c157775d1d5a68d4d48be3828a2320f3ac941b
Sep 16 12:15:27.008253 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:27.008216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp" event={"ID":"03188b71-c35e-48cc-a837-6d024c966f4b","Type":"ContainerStarted","Data":"24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095"}
Sep 16 12:15:27.008460 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:27.008259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp" event={"ID":"03188b71-c35e-48cc-a837-6d024c966f4b","Type":"ContainerStarted","Data":"ba185de0063f5ebf17a1782e74c157775d1d5a68d4d48be3828a2320f3ac941b"}
Sep 16 12:15:27.025532 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:27.025464    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp" podStartSLOduration=1.025449764 podStartE2EDuration="1.025449764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:27.02416453 +0000 UTC m=+816.661489367" watchObservedRunningTime="2026-09-16 12:15:27.025449764 +0000 UTC m=+816.662774604"
Sep 16 12:15:36.353045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:36.353007    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:36.353451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:36.353102    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:36.365701 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:36.365674    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:37.092623 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:37.092597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:15:39.777913 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:39.777880    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"]
Sep 16 12:15:39.778354 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:39.778131    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn" podUID="893ad837-39fd-4cab-b72a-f1a4dd7129e9" containerName="maas-api" containerID="cri-o://d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34" gracePeriod=30
Sep 16 12:15:40.034293 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.034231    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:40.102392 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.102348    2578 generic.go:356] "Generic (PLEG): container finished" podID="893ad837-39fd-4cab-b72a-f1a4dd7129e9" containerID="d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34" exitCode=0
Sep 16 12:15:40.102392 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.102394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"
Sep 16 12:15:40.102648 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.102412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn" event={"ID":"893ad837-39fd-4cab-b72a-f1a4dd7129e9","Type":"ContainerDied","Data":"d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34"}
Sep 16 12:15:40.102648 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.102532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn" event={"ID":"893ad837-39fd-4cab-b72a-f1a4dd7129e9","Type":"ContainerDied","Data":"5f16e1db152098bd12e9a586c74082cc963f124eb1e32a43da151d5921af9e8b"}
Sep 16 12:15:40.102648 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.102567    2578 scope.go:117] "RemoveContainer" containerID="d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34"
Sep 16 12:15:40.112775 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.112753    2578 scope.go:117] "RemoveContainer" containerID="d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34"
Sep 16 12:15:40.113112 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:15:40.113091    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34\": container with ID starting with d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34 not found: ID does not exist" containerID="d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34"
Sep 16 12:15:40.113171 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.113122    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34"} err="failed to get container status \"d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34\": rpc error: code = NotFound desc = could not find container \"d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34\": container with ID starting with d6df0b7a3702cc799348e10fcfd502c890a1b458641ace0aa893426d6c7f1e34 not found: ID does not exist"
Sep 16 12:15:40.137684 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.137648    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-metrics-certs\") pod \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") "
Sep 16 12:15:40.137837 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.137735    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls\") pod \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") "
Sep 16 12:15:40.137837 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.137778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893ad837-39fd-4cab-b72a-f1a4dd7129e9-openshift-service-ca\") pod \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") "
Sep 16 12:15:40.137837 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.137798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7zbb\" (UniqueName: \"kubernetes.io/projected/893ad837-39fd-4cab-b72a-f1a4dd7129e9-kube-api-access-z7zbb\") pod \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\" (UID: \"893ad837-39fd-4cab-b72a-f1a4dd7129e9\") "
Sep 16 12:15:40.138132 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.138109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893ad837-39fd-4cab-b72a-f1a4dd7129e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "893ad837-39fd-4cab-b72a-f1a4dd7129e9" (UID: "893ad837-39fd-4cab-b72a-f1a4dd7129e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:40.140103 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.140072    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893ad837-39fd-4cab-b72a-f1a4dd7129e9-kube-api-access-z7zbb" (OuterVolumeSpecName: "kube-api-access-z7zbb") pod "893ad837-39fd-4cab-b72a-f1a4dd7129e9" (UID: "893ad837-39fd-4cab-b72a-f1a4dd7129e9"). InnerVolumeSpecName "kube-api-access-z7zbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:40.140226 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.140121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "893ad837-39fd-4cab-b72a-f1a4dd7129e9" (UID: "893ad837-39fd-4cab-b72a-f1a4dd7129e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:15:40.140226 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.140136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "893ad837-39fd-4cab-b72a-f1a4dd7129e9" (UID: "893ad837-39fd-4cab-b72a-f1a4dd7129e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:15:40.239117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.239067    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:40.239117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.239113    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893ad837-39fd-4cab-b72a-f1a4dd7129e9-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:40.239117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.239122    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893ad837-39fd-4cab-b72a-f1a4dd7129e9-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:40.239117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.239132    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zbb\" (UniqueName: \"kubernetes.io/projected/893ad837-39fd-4cab-b72a-f1a4dd7129e9-kube-api-access-z7zbb\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:15:40.424945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.424911    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"]
Sep 16 12:15:40.428174 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.428146    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f74dc7e1-864dfd87b8-p92kn"]
Sep 16 12:15:40.884996 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:40.884964    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="893ad837-39fd-4cab-b72a-f1a4dd7129e9" path="/var/lib/kubelet/pods/893ad837-39fd-4cab-b72a-f1a4dd7129e9/volumes"
Sep 16 12:15:44.848368 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.848335    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"]
Sep 16 12:15:44.848862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.848754    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="893ad837-39fd-4cab-b72a-f1a4dd7129e9" containerName="maas-api"
Sep 16 12:15:44.848862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.848766    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="893ad837-39fd-4cab-b72a-f1a4dd7129e9" containerName="maas-api"
Sep 16 12:15:44.848862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.848830    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="893ad837-39fd-4cab-b72a-f1a4dd7129e9" containerName="maas-api"
Sep 16 12:15:44.851862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.851840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.854291 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.854269    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3ad35c8a-serving-cert\""
Sep 16 12:15:44.862549 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.862504    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"]
Sep 16 12:15:44.883367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.883330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfr7b\" (UniqueName: \"kubernetes.io/projected/4c99c307-179e-4dde-864b-40809ea6c609-kube-api-access-xfr7b\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.883574 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.883452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-metrics-certs\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.883574 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.883499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c99c307-179e-4dde-864b-40809ea6c609-openshift-service-ca\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.883705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.883584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.984341 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.984296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfr7b\" (UniqueName: \"kubernetes.io/projected/4c99c307-179e-4dde-864b-40809ea6c609-kube-api-access-xfr7b\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.984570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.984355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-metrics-certs\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.984570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.984381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c99c307-179e-4dde-864b-40809ea6c609-openshift-service-ca\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.984570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.984405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.984570 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:15:44.984567    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-serving-cert: secret "maas-api-e2e-ait-3ad35c8a-serving-cert" not found
Sep 16 12:15:44.984744 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:15:44.984637    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls podName:4c99c307-179e-4dde-864b-40809ea6c609 nodeName:}" failed. No retries permitted until 2026-09-16 12:15:45.484617506 +0000 UTC m=+835.121942322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls") pod "maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz" (UID: "4c99c307-179e-4dde-864b-40809ea6c609") : secret "maas-api-e2e-ait-3ad35c8a-serving-cert" not found
Sep 16 12:15:44.985153 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.985128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c99c307-179e-4dde-864b-40809ea6c609-openshift-service-ca\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.987084 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.987061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-metrics-certs\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:44.994194 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:44.994171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfr7b\" (UniqueName: \"kubernetes.io/projected/4c99c307-179e-4dde-864b-40809ea6c609-kube-api-access-xfr7b\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:45.487999 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:45.487954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:45.490543 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:45.490502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls\") pod \"maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:45.764495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:45.764410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:46.036254 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:46.036223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"]
Sep 16 12:15:46.037162 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:15:46.037135    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c99c307_179e_4dde_864b_40809ea6c609.slice/crio-4673fd93260defbe30c775c170f18370faec27d0361ff8ac49f5ef8beddadfe6 WatchSource:0}: Error finding container 4673fd93260defbe30c775c170f18370faec27d0361ff8ac49f5ef8beddadfe6: Status 404 returned error can't find the container with id 4673fd93260defbe30c775c170f18370faec27d0361ff8ac49f5ef8beddadfe6
Sep 16 12:15:46.137321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:46.137286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz" event={"ID":"4c99c307-179e-4dde-864b-40809ea6c609","Type":"ContainerStarted","Data":"4673fd93260defbe30c775c170f18370faec27d0361ff8ac49f5ef8beddadfe6"}
Sep 16 12:15:47.144826 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.144721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz" event={"ID":"4c99c307-179e-4dde-864b-40809ea6c609","Type":"ContainerStarted","Data":"cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c"}
Sep 16 12:15:47.144826 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.144792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:47.164739 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.164688    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz" podStartSLOduration=2.392246478 podStartE2EDuration="3.164669834s" totalImagesPullingTime="772.423356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:47.162764644 +0000 UTC m=+836.800089485" watchObservedRunningTime="2026-09-16 12:15:47.164669834 +0000 UTC m=+836.801994671"
Sep 16 12:15:47.638892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.638855    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"]
Sep 16 12:15:47.642505 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.642485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.646597 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.646567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-641983\"/\"openshift-service-ca.crt\""
Sep 16 12:15:47.646597 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.646584    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-641983\"/\"default-dockercfg-6bvgh\""
Sep 16 12:15:47.646597 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.646592    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-641983\"/\"kube-root-ca.crt\""
Sep 16 12:15:47.646810 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.646573    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-641983\"/\"test-model-641983-kserve-self-signed-certs\""
Sep 16 12:15:47.651380 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.651348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"]
Sep 16 12:15:47.710493 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.710456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7f9x\" (UniqueName: \"kubernetes.io/projected/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-kube-api-access-h7f9x\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.710710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.710561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-dshm\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.710710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.710582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tmp-dir\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.710710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.710598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tls-certs\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.710710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.710617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-model-cache\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.710710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.710660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-home\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.811941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.811898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tmp-dir\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.811941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.811937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tls-certs\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.812200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.811959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-model-cache\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.812200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.811989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-home\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.812200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.812063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7f9x\" (UniqueName: \"kubernetes.io/projected/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-kube-api-access-h7f9x\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.812200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.812151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-dshm\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.812376 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.812351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tmp-dir\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.812464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.812384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-model-cache\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.812464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.812405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-home\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.814586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.814564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-dshm\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.814837 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.814820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tls-certs\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.827486 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.827451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7f9x\" (UniqueName: \"kubernetes.io/projected/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-kube-api-access-h7f9x\") pod \"test-model-641983-kserve-766d6b5fc9-zr8k8\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") " pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:47.952800 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:47.952683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:48.229893 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:48.229864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"]
Sep 16 12:15:48.231277 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:15:48.231240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd95f44c5_4f1e_4e3a_b862_fc9c5cc9034f.slice/crio-238d7209f80f71ec82f50a9bfa8ec895dd651024e027b83796ee9d3640678971 WatchSource:0}: Error finding container 238d7209f80f71ec82f50a9bfa8ec895dd651024e027b83796ee9d3640678971: Status 404 returned error can't find the container with id 238d7209f80f71ec82f50a9bfa8ec895dd651024e027b83796ee9d3640678971
Sep 16 12:15:49.157425 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:49.157389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8" event={"ID":"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f","Type":"ContainerStarted","Data":"fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381"}
Sep 16 12:15:49.157724 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:49.157432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8" event={"ID":"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f","Type":"ContainerStarted","Data":"238d7209f80f71ec82f50a9bfa8ec895dd651024e027b83796ee9d3640678971"}
Sep 16 12:15:49.176048 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:49.175992    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8" podStartSLOduration=2.175976144 podStartE2EDuration="2.175976144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:49.173452529 +0000 UTC m=+838.810777368" watchObservedRunningTime="2026-09-16 12:15:49.175976144 +0000 UTC m=+838.813300981"
Sep 16 12:15:53.156135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:53.156100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:15:57.953575 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:57.953535    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:57.953575 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:57.953579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:57.965922 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:57.965892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:15:58.227339 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:15:58.227259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:16:10.137293 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.137257    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"]
Sep 16 12:16:10.142491 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.142474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.147047 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.146943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-dc3e2c\"/\"openshift-service-ca.crt\""
Sep 16 12:16:10.147980 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.147953    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-dc3e2c\"/\"kube-root-ca.crt\""
Sep 16 12:16:10.148112 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.148038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-dc3e2c\"/\"autores-model-dc3e2c-kserve-self-signed-certs\""
Sep 16 12:16:10.148891 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.148868    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-dc3e2c\"/\"default-dockercfg-krxpq\""
Sep 16 12:16:10.150875 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.150713    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"]
Sep 16 12:16:10.326135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.326102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-tmp-dir\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.326329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.326142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-model-cache\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.326329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.326184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-home\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.326329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.326265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-dshm\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.326329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.326313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde26748-247a-45aa-8378-e50d6580f348-tls-certs\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.326501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.326356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5hm\" (UniqueName: \"kubernetes.io/projected/fde26748-247a-45aa-8378-e50d6580f348-kube-api-access-bs5hm\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.427670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.427575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5hm\" (UniqueName: \"kubernetes.io/projected/fde26748-247a-45aa-8378-e50d6580f348-kube-api-access-bs5hm\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.427670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.427649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-tmp-dir\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.427670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.427667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-model-cache\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.427970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.427826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-home\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.427970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.427877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-dshm\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.427970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.427934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde26748-247a-45aa-8378-e50d6580f348-tls-certs\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.428123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.428086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-model-cache\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.428123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.428102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-tmp-dir\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.428273 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.428149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-home\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.430432 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.430410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-dshm\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.430564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.430549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde26748-247a-45aa-8378-e50d6580f348-tls-certs\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.440308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.440279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5hm\" (UniqueName: \"kubernetes.io/projected/fde26748-247a-45aa-8378-e50d6580f348-kube-api-access-bs5hm\") pod \"autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") " pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.455021 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.454986    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:10.731698 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:10.730191    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"]
Sep 16 12:16:10.732338 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:16:10.732309    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde26748_247a_45aa_8378_e50d6580f348.slice/crio-f9cb43a53ffbde4a2fb693651be4f25001f05a26163ede02e3d66be4627d7d03 WatchSource:0}: Error finding container f9cb43a53ffbde4a2fb693651be4f25001f05a26163ede02e3d66be4627d7d03: Status 404 returned error can't find the container with id f9cb43a53ffbde4a2fb693651be4f25001f05a26163ede02e3d66be4627d7d03
Sep 16 12:16:11.288833 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:11.288799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw" event={"ID":"fde26748-247a-45aa-8378-e50d6580f348","Type":"ContainerStarted","Data":"8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f"}
Sep 16 12:16:11.288833 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:11.288841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw" event={"ID":"fde26748-247a-45aa-8378-e50d6580f348","Type":"ContainerStarted","Data":"f9cb43a53ffbde4a2fb693651be4f25001f05a26163ede02e3d66be4627d7d03"}
Sep 16 12:16:11.306700 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:11.306648    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw" podStartSLOduration=1.306629631 podStartE2EDuration="1.306629631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:11.305005418 +0000 UTC m=+860.942330255" watchObservedRunningTime="2026-09-16 12:16:11.306629631 +0000 UTC m=+860.943954468"
Sep 16 12:16:14.983306 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:14.983275    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"]
Sep 16 12:16:14.983778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:14.983523    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz" podUID="4c99c307-179e-4dde-864b-40809ea6c609" containerName="maas-api" containerID="cri-o://cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c" gracePeriod=30
Sep 16 12:16:15.227214 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.227189    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:16:15.278862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.278788    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c99c307-179e-4dde-864b-40809ea6c609-openshift-service-ca\") pod \"4c99c307-179e-4dde-864b-40809ea6c609\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") "
Sep 16 12:16:15.278862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.278827    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-metrics-certs\") pod \"4c99c307-179e-4dde-864b-40809ea6c609\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") "
Sep 16 12:16:15.279060 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.278864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfr7b\" (UniqueName: \"kubernetes.io/projected/4c99c307-179e-4dde-864b-40809ea6c609-kube-api-access-xfr7b\") pod \"4c99c307-179e-4dde-864b-40809ea6c609\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") "
Sep 16 12:16:15.279060 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.278906    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls\") pod \"4c99c307-179e-4dde-864b-40809ea6c609\" (UID: \"4c99c307-179e-4dde-864b-40809ea6c609\") "
Sep 16 12:16:15.279219 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.279188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c99c307-179e-4dde-864b-40809ea6c609-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4c99c307-179e-4dde-864b-40809ea6c609" (UID: "4c99c307-179e-4dde-864b-40809ea6c609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:16:15.281210 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.281179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c99c307-179e-4dde-864b-40809ea6c609-kube-api-access-xfr7b" (OuterVolumeSpecName: "kube-api-access-xfr7b") pod "4c99c307-179e-4dde-864b-40809ea6c609" (UID: "4c99c307-179e-4dde-864b-40809ea6c609"). InnerVolumeSpecName "kube-api-access-xfr7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:15.281210 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.281195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4c99c307-179e-4dde-864b-40809ea6c609" (UID: "4c99c307-179e-4dde-864b-40809ea6c609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:16:15.281407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.281209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4c99c307-179e-4dde-864b-40809ea6c609" (UID: "4c99c307-179e-4dde-864b-40809ea6c609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:16:15.311600 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.311566    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c99c307-179e-4dde-864b-40809ea6c609" containerID="cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c" exitCode=0
Sep 16 12:16:15.311786 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.311619    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"
Sep 16 12:16:15.311786 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.311639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz" event={"ID":"4c99c307-179e-4dde-864b-40809ea6c609","Type":"ContainerDied","Data":"cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c"}
Sep 16 12:16:15.311786 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.311682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz" event={"ID":"4c99c307-179e-4dde-864b-40809ea6c609","Type":"ContainerDied","Data":"4673fd93260defbe30c775c170f18370faec27d0361ff8ac49f5ef8beddadfe6"}
Sep 16 12:16:15.311786 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.311705    2578 scope.go:117] "RemoveContainer" containerID="cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c"
Sep 16 12:16:15.322208 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.322191    2578 scope.go:117] "RemoveContainer" containerID="cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c"
Sep 16 12:16:15.322501 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:16:15.322481    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c\": container with ID starting with cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c not found: ID does not exist" containerID="cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c"
Sep 16 12:16:15.322567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.322527    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c"} err="failed to get container status \"cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c\": rpc error: code = NotFound desc = could not find container \"cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c\": container with ID starting with cacbae5926da328792c9e630c3ac410df83c278eaa4e9ee7121ad5179108099c not found: ID does not exist"
Sep 16 12:16:15.336279 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.336245    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"]
Sep 16 12:16:15.340084 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.340057    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ad35c8a-5d56dfd7c8-7qkhz"]
Sep 16 12:16:15.380045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.380011    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfr7b\" (UniqueName: \"kubernetes.io/projected/4c99c307-179e-4dde-864b-40809ea6c609-kube-api-access-xfr7b\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:16:15.380045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.380040    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:16:15.380045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.380049    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c99c307-179e-4dde-864b-40809ea6c609-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:16:15.380045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:15.380057    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c99c307-179e-4dde-864b-40809ea6c609-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:16:16.885260 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:16.885227    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c99c307-179e-4dde-864b-40809ea6c609" path="/var/lib/kubelet/pods/4c99c307-179e-4dde-864b-40809ea6c609/volumes"
Sep 16 12:16:20.455846 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:20.455809    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:20.455846 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:20.455858    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:20.468554 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:20.468506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:21.358428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:21.358396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:16:46.455313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:46.455234    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"]
Sep 16 12:16:50.858264 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:50.858232    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:16:50.858739 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:50.858339    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:16:55.579972 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.579937    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"]
Sep 16 12:16:55.580602 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.580191    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn" podUID="16d0498f-5e83-4ac7-b34c-d33928938b15" containerName="maas-api" containerID="cri-o://f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2" gracePeriod=30
Sep 16 12:16:55.825786 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.825761    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:16:55.869814 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.869737    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-metrics-certs\") pod \"16d0498f-5e83-4ac7-b34c-d33928938b15\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") "
Sep 16 12:16:55.869814 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.869771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d0498f-5e83-4ac7-b34c-d33928938b15-openshift-service-ca\") pod \"16d0498f-5e83-4ac7-b34c-d33928938b15\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") "
Sep 16 12:16:55.870031 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.869909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr7pg\" (UniqueName: \"kubernetes.io/projected/16d0498f-5e83-4ac7-b34c-d33928938b15-kube-api-access-dr7pg\") pod \"16d0498f-5e83-4ac7-b34c-d33928938b15\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") "
Sep 16 12:16:55.870031 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.870005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls\") pod \"16d0498f-5e83-4ac7-b34c-d33928938b15\" (UID: \"16d0498f-5e83-4ac7-b34c-d33928938b15\") "
Sep 16 12:16:55.870189 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.870158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d0498f-5e83-4ac7-b34c-d33928938b15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16d0498f-5e83-4ac7-b34c-d33928938b15" (UID: "16d0498f-5e83-4ac7-b34c-d33928938b15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:16:55.870374 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.870346    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d0498f-5e83-4ac7-b34c-d33928938b15-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:16:55.872166 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.872135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d0498f-5e83-4ac7-b34c-d33928938b15-kube-api-access-dr7pg" (OuterVolumeSpecName: "kube-api-access-dr7pg") pod "16d0498f-5e83-4ac7-b34c-d33928938b15" (UID: "16d0498f-5e83-4ac7-b34c-d33928938b15"). InnerVolumeSpecName "kube-api-access-dr7pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:55.872262 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.872160    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16d0498f-5e83-4ac7-b34c-d33928938b15" (UID: "16d0498f-5e83-4ac7-b34c-d33928938b15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:16:55.872327 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.872285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16d0498f-5e83-4ac7-b34c-d33928938b15" (UID: "16d0498f-5e83-4ac7-b34c-d33928938b15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:16:55.971528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.971481    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr7pg\" (UniqueName: \"kubernetes.io/projected/16d0498f-5e83-4ac7-b34c-d33928938b15-kube-api-access-dr7pg\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:16:55.971528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.971504    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:16:55.971765 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:55.971540    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d0498f-5e83-4ac7-b34c-d33928938b15-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:16:56.543831 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.543795    2578 generic.go:356] "Generic (PLEG): container finished" podID="16d0498f-5e83-4ac7-b34c-d33928938b15" containerID="f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2" exitCode=0
Sep 16 12:16:56.544033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.543843    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"
Sep 16 12:16:56.544033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.543890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn" event={"ID":"16d0498f-5e83-4ac7-b34c-d33928938b15","Type":"ContainerDied","Data":"f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2"}
Sep 16 12:16:56.544033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.543950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn" event={"ID":"16d0498f-5e83-4ac7-b34c-d33928938b15","Type":"ContainerDied","Data":"b04eaadbf2cb38ad06cb5d7acf2773f4569786077a012140939de4ca2b9563ae"}
Sep 16 12:16:56.544033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.543980    2578 scope.go:117] "RemoveContainer" containerID="f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2"
Sep 16 12:16:56.554586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.554565    2578 scope.go:117] "RemoveContainer" containerID="f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2"
Sep 16 12:16:56.554882 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:16:56.554862    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2\": container with ID starting with f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2 not found: ID does not exist" containerID="f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2"
Sep 16 12:16:56.554954 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.554890    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2"} err="failed to get container status \"f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2\": rpc error: code = NotFound desc = could not find container \"f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2\": container with ID starting with f6e3d398a0fc7a290b0c556de789aad6d351565718754dd322707d26f9a2def2 not found: ID does not exist"
Sep 16 12:16:56.565463 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.565442    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"]
Sep 16 12:16:56.569263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.569241    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc3e2c-67fccb9d99-ttcjn"]
Sep 16 12:16:56.739343 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.739310    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"]
Sep 16 12:16:56.739759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.739710    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c99c307-179e-4dde-864b-40809ea6c609" containerName="maas-api"
Sep 16 12:16:56.739759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.739722    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c99c307-179e-4dde-864b-40809ea6c609" containerName="maas-api"
Sep 16 12:16:56.739759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.739735    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d0498f-5e83-4ac7-b34c-d33928938b15" containerName="maas-api"
Sep 16 12:16:56.739759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.739743    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d0498f-5e83-4ac7-b34c-d33928938b15" containerName="maas-api"
Sep 16 12:16:56.739898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.739815    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c99c307-179e-4dde-864b-40809ea6c609" containerName="maas-api"
Sep 16 12:16:56.739898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.739832    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d0498f-5e83-4ac7-b34c-d33928938b15" containerName="maas-api"
Sep 16 12:16:56.744368 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.744352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.747112 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.747090    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"premium-w4-556077-kserve-self-signed-certs\""
Sep 16 12:16:56.753963 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.753941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"]
Sep 16 12:16:56.778281 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.778255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4h7\" (UniqueName: \"kubernetes.io/projected/3852cef5-bbbf-4f0e-a746-0f44164cebeb-kube-api-access-jf4h7\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.778413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.778306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-model-cache\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.778413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.778366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-dshm\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.778538 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.778425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-home\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.778538 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.778447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tmp-dir\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.778651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.778549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tls-certs\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-model-cache\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-dshm\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-home\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tmp-dir\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tls-certs\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4h7\" (UniqueName: \"kubernetes.io/projected/3852cef5-bbbf-4f0e-a746-0f44164cebeb-kube-api-access-jf4h7\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-model-cache\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-home\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.879646 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.879631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tmp-dir\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.881507 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.881486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-dshm\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.881806 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.881787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tls-certs\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.891789 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.891758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4h7\" (UniqueName: \"kubernetes.io/projected/3852cef5-bbbf-4f0e-a746-0f44164cebeb-kube-api-access-jf4h7\") pod \"premium-w4-556077-kserve-66fbf947b6-x5j9j\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") " pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:56.894912 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:56.894887    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16d0498f-5e83-4ac7-b34c-d33928938b15" path="/var/lib/kubelet/pods/16d0498f-5e83-4ac7-b34c-d33928938b15/volumes"
Sep 16 12:16:57.054301 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:57.054260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:16:57.326241 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:57.326210    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"]
Sep 16 12:16:57.327220 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:16:57.327189    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3852cef5_bbbf_4f0e_a746_0f44164cebeb.slice/crio-5c20798b5aaa5ea937d3857bb884b836a5bbcae32c5541d3a22397ad7f0e87e4 WatchSource:0}: Error finding container 5c20798b5aaa5ea937d3857bb884b836a5bbcae32c5541d3a22397ad7f0e87e4: Status 404 returned error can't find the container with id 5c20798b5aaa5ea937d3857bb884b836a5bbcae32c5541d3a22397ad7f0e87e4
Sep 16 12:16:57.552004 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:57.551966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j" event={"ID":"3852cef5-bbbf-4f0e-a746-0f44164cebeb","Type":"ContainerStarted","Data":"6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2"}
Sep 16 12:16:57.552004 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:57.552007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j" event={"ID":"3852cef5-bbbf-4f0e-a746-0f44164cebeb","Type":"ContainerStarted","Data":"5c20798b5aaa5ea937d3857bb884b836a5bbcae32c5541d3a22397ad7f0e87e4"}
Sep 16 12:16:57.570553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:16:57.570484    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j" podStartSLOduration=1.570469273 podStartE2EDuration="1.570469273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:57.567346118 +0000 UTC m=+907.204670956" watchObservedRunningTime="2026-09-16 12:16:57.570469273 +0000 UTC m=+907.207794110"
Sep 16 12:17:01.478648 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.478604    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw" podUID="fde26748-247a-45aa-8378-e50d6580f348" containerName="main" containerID="cri-o://8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f" gracePeriod=45
Sep 16 12:17:01.729847 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.729781    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:17:01.827350 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827314    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-tmp-dir\") pod \"fde26748-247a-45aa-8378-e50d6580f348\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") "
Sep 16 12:17:01.827564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827359    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-home\") pod \"fde26748-247a-45aa-8378-e50d6580f348\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") "
Sep 16 12:17:01.827564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs5hm\" (UniqueName: \"kubernetes.io/projected/fde26748-247a-45aa-8378-e50d6580f348-kube-api-access-bs5hm\") pod \"fde26748-247a-45aa-8378-e50d6580f348\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") "
Sep 16 12:17:01.827564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827436    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-model-cache\") pod \"fde26748-247a-45aa-8378-e50d6580f348\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") "
Sep 16 12:17:01.827564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-dshm\") pod \"fde26748-247a-45aa-8378-e50d6580f348\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") "
Sep 16 12:17:01.827564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827490    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde26748-247a-45aa-8378-e50d6580f348-tls-certs\") pod \"fde26748-247a-45aa-8378-e50d6580f348\" (UID: \"fde26748-247a-45aa-8378-e50d6580f348\") "
Sep 16 12:17:01.827836 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fde26748-247a-45aa-8378-e50d6580f348" (UID: "fde26748-247a-45aa-8378-e50d6580f348"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:01.827836 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-home" (OuterVolumeSpecName: "home") pod "fde26748-247a-45aa-8378-e50d6580f348" (UID: "fde26748-247a-45aa-8378-e50d6580f348"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:01.827836 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827746    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-model-cache" (OuterVolumeSpecName: "model-cache") pod "fde26748-247a-45aa-8378-e50d6580f348" (UID: "fde26748-247a-45aa-8378-e50d6580f348"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:01.827940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827854    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.827940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827873    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.827940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.827913    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.829859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.829836    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde26748-247a-45aa-8378-e50d6580f348-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fde26748-247a-45aa-8378-e50d6580f348" (UID: "fde26748-247a-45aa-8378-e50d6580f348"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:17:01.829945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.829925    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-dshm" (OuterVolumeSpecName: "dshm") pod "fde26748-247a-45aa-8378-e50d6580f348" (UID: "fde26748-247a-45aa-8378-e50d6580f348"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:01.829995 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.829979    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde26748-247a-45aa-8378-e50d6580f348-kube-api-access-bs5hm" (OuterVolumeSpecName: "kube-api-access-bs5hm") pod "fde26748-247a-45aa-8378-e50d6580f348" (UID: "fde26748-247a-45aa-8378-e50d6580f348"). InnerVolumeSpecName "kube-api-access-bs5hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:17:01.928988 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.928950    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde26748-247a-45aa-8378-e50d6580f348-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.928988 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.928985    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde26748-247a-45aa-8378-e50d6580f348-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.929265 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:01.929001    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs5hm\" (UniqueName: \"kubernetes.io/projected/fde26748-247a-45aa-8378-e50d6580f348-kube-api-access-bs5hm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:17:02.581162 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.581131    2578 generic.go:356] "Generic (PLEG): container finished" podID="fde26748-247a-45aa-8378-e50d6580f348" containerID="8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f" exitCode=0
Sep 16 12:17:02.581609 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.581178    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"
Sep 16 12:17:02.581609 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.581214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw" event={"ID":"fde26748-247a-45aa-8378-e50d6580f348","Type":"ContainerDied","Data":"8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f"}
Sep 16 12:17:02.581609 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.581273    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw" event={"ID":"fde26748-247a-45aa-8378-e50d6580f348","Type":"ContainerDied","Data":"f9cb43a53ffbde4a2fb693651be4f25001f05a26163ede02e3d66be4627d7d03"}
Sep 16 12:17:02.581609 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.581296    2578 scope.go:117] "RemoveContainer" containerID="8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f"
Sep 16 12:17:02.591468 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.591447    2578 scope.go:117] "RemoveContainer" containerID="8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f"
Sep 16 12:17:02.591781 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:17:02.591755    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f\": container with ID starting with 8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f not found: ID does not exist" containerID="8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f"
Sep 16 12:17:02.591849 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.591790    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f"} err="failed to get container status \"8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f\": rpc error: code = NotFound desc = could not find container \"8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f\": container with ID starting with 8e1a69d615f817236436f937bb049f0025e60077e03f30ec1fedf1d985f1ea3f not found: ID does not exist"
Sep 16 12:17:02.610147 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.610125    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"]
Sep 16 12:17:02.614358 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.614337    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-dc3e2c/autores-model-dc3e2c-kserve-6bcfb76dcf-c7spw"]
Sep 16 12:17:02.884828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:02.884748    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fde26748-247a-45aa-8378-e50d6580f348" path="/var/lib/kubelet/pods/fde26748-247a-45aa-8378-e50d6580f348/volumes"
Sep 16 12:17:07.055066 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:07.055033    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:17:07.055066 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:07.055076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:17:07.067807 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:07.067780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:17:07.619107 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:07.619083    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:17:49.255963 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.255929    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"]
Sep 16 12:17:49.256418 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.256280    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde26748-247a-45aa-8378-e50d6580f348" containerName="main"
Sep 16 12:17:49.256418 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.256290    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde26748-247a-45aa-8378-e50d6580f348" containerName="main"
Sep 16 12:17:49.256418 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.256364    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde26748-247a-45aa-8378-e50d6580f348" containerName="main"
Sep 16 12:17:49.259412 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.259396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.262581 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.262559    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-08170a\"/\"kube-root-ca.crt\""
Sep 16 12:17:49.263728 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.263709    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-08170a\"/\"test-model-08170a-kserve-self-signed-certs\""
Sep 16 12:17:49.263828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.263744    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-08170a\"/\"openshift-service-ca.crt\""
Sep 16 12:17:49.263828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.263765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-08170a\"/\"default-dockercfg-fcs6z\""
Sep 16 12:17:49.267907 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.267888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"]
Sep 16 12:17:49.365139 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.365110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s77w\" (UniqueName: \"kubernetes.io/projected/82faedf9-2bc6-47f4-a298-bb056e2fa89a-kube-api-access-4s77w\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.365318 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.365144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tmp-dir\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.365318 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.365189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-model-cache\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.365318 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.365233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-home\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.365318 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.365267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tls-certs\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.365462 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.365348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-dshm\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466042 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-dshm\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466223 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s77w\" (UniqueName: \"kubernetes.io/projected/82faedf9-2bc6-47f4-a298-bb056e2fa89a-kube-api-access-4s77w\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466223 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tmp-dir\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466223 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-model-cache\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466223 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-home\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466223 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tls-certs\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466598 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-model-cache\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466598 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-home\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.466598 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.466594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tmp-dir\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.468386 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.468367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-dshm\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.468776 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.468757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tls-certs\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.473802 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.473781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s77w\" (UniqueName: \"kubernetes.io/projected/82faedf9-2bc6-47f4-a298-bb056e2fa89a-kube-api-access-4s77w\") pod \"test-model-08170a-kserve-b67c47ddc-gls2z\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") " pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.570041 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.569958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:49.843599 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:49.843574    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"]
Sep 16 12:17:49.845815 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:17:49.845787    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82faedf9_2bc6_47f4_a298_bb056e2fa89a.slice/crio-9a117aa37699f92b3368f24943b48cd81c8767800c5008d74d1e8addc9d9e8a5 WatchSource:0}: Error finding container 9a117aa37699f92b3368f24943b48cd81c8767800c5008d74d1e8addc9d9e8a5: Status 404 returned error can't find the container with id 9a117aa37699f92b3368f24943b48cd81c8767800c5008d74d1e8addc9d9e8a5
Sep 16 12:17:50.856775 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:50.856740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z" event={"ID":"82faedf9-2bc6-47f4-a298-bb056e2fa89a","Type":"ContainerStarted","Data":"2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea"}
Sep 16 12:17:50.856775 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:50.856785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z" event={"ID":"82faedf9-2bc6-47f4-a298-bb056e2fa89a","Type":"ContainerStarted","Data":"9a117aa37699f92b3368f24943b48cd81c8767800c5008d74d1e8addc9d9e8a5"}
Sep 16 12:17:50.881724 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:50.881680    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z" podStartSLOduration=1.881663444 podStartE2EDuration="1.881663444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:50.878001485 +0000 UTC m=+960.515326323" watchObservedRunningTime="2026-09-16 12:17:50.881663444 +0000 UTC m=+960.518988279"
Sep 16 12:17:59.571098 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:59.571067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:59.571491 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:59.571129    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:59.583603 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:59.583578    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:17:59.916096 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:17:59.916002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:18:31.784649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.784609    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"]
Sep 16 12:18:31.788216 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.788199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.791632 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.791611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7810a5\"/\"kube-root-ca.crt\""
Sep 16 12:18:31.792428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.792412    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7810a5\"/\"openshift-service-ca.crt\""
Sep 16 12:18:31.792782 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.792767    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7810a5\"/\"default-dockercfg-28q5t\""
Sep 16 12:18:31.797726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.797706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7810a5\"/\"explicit-model-7810a5-kserve-self-signed-certs\""
Sep 16 12:18:31.813727 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.813697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"]
Sep 16 12:18:31.869675 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.869640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfvf\" (UniqueName: \"kubernetes.io/projected/a6f4c1a5-a084-4208-91c7-670ee6307503-kube-api-access-bjfvf\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.869874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.869686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-model-cache\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.869874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.869807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-home\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.869874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.869842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f4c1a5-a084-4208-91c7-670ee6307503-tls-certs\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.870047 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.869919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-tmp-dir\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.870047 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.869985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-dshm\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.970991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.970959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-tmp-dir\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.971171 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.971005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-dshm\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.971171 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.971049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfvf\" (UniqueName: \"kubernetes.io/projected/a6f4c1a5-a084-4208-91c7-670ee6307503-kube-api-access-bjfvf\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.971171 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.971075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-model-cache\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.971171 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.971131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-home\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.971171 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.971153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f4c1a5-a084-4208-91c7-670ee6307503-tls-certs\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.971452 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.971431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-tmp-dir\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.971546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.971501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-home\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.971593 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.971567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-model-cache\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.973623 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.973599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-dshm\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.973744 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.973728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f4c1a5-a084-4208-91c7-670ee6307503-tls-certs\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:31.978873 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:31.978855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfvf\" (UniqueName: \"kubernetes.io/projected/a6f4c1a5-a084-4208-91c7-670ee6307503-kube-api-access-bjfvf\") pod \"explicit-model-7810a5-kserve-cf44c9c5c-sdk7v\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") " pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:32.097840 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:32.097762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:32.373870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:32.373840    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"]
Sep 16 12:18:32.375566 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:18:32.375540    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6f4c1a5_a084_4208_91c7_670ee6307503.slice/crio-4c6fb889500d211740cb2e75d92ff962e8ee1917c50eb40fc5c10d3a8a4cda69 WatchSource:0}: Error finding container 4c6fb889500d211740cb2e75d92ff962e8ee1917c50eb40fc5c10d3a8a4cda69: Status 404 returned error can't find the container with id 4c6fb889500d211740cb2e75d92ff962e8ee1917c50eb40fc5c10d3a8a4cda69
Sep 16 12:18:33.084286 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:33.084254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v" event={"ID":"a6f4c1a5-a084-4208-91c7-670ee6307503","Type":"ContainerStarted","Data":"55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440"}
Sep 16 12:18:33.084286 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:33.084292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v" event={"ID":"a6f4c1a5-a084-4208-91c7-670ee6307503","Type":"ContainerStarted","Data":"4c6fb889500d211740cb2e75d92ff962e8ee1917c50eb40fc5c10d3a8a4cda69"}
Sep 16 12:18:33.101743 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:33.101696    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v" podStartSLOduration=2.101678756 podStartE2EDuration="2.101678756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:18:33.099879045 +0000 UTC m=+1002.737203910" watchObservedRunningTime="2026-09-16 12:18:33.101678756 +0000 UTC m=+1002.739003616"
Sep 16 12:18:42.098226 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:42.098192    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:42.098739 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:42.098350    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:42.111146 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:42.111124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:42.143853 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:42.143818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:18:48.348601 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.348568    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"]
Sep 16 12:18:48.352187 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.352164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.354945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.354917    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f916d574-serving-cert\""
Sep 16 12:18:48.365252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.365227    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"]
Sep 16 12:18:48.435631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.435595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.435785 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.435745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-metrics-certs\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.435857 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.435810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfvl\" (UniqueName: \"kubernetes.io/projected/e01c7126-4436-4844-babb-9810882174f2-kube-api-access-6tfvl\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.435914 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.435869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7126-4436-4844-babb-9810882174f2-openshift-service-ca\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.537054 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.537016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7126-4436-4844-babb-9810882174f2-openshift-service-ca\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.537054 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.537063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.537297 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.537148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-metrics-certs\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.537297 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.537211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfvl\" (UniqueName: \"kubernetes.io/projected/e01c7126-4436-4844-babb-9810882174f2-kube-api-access-6tfvl\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.537297 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:18:48.537235    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-serving-cert: secret "maas-api-e2e-mt-f916d574-serving-cert" not found
Sep 16 12:18:48.537473 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:18:48.537313    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls podName:e01c7126-4436-4844-babb-9810882174f2 nodeName:}" failed. No retries permitted until 2026-09-16 12:18:49.037289795 +0000 UTC m=+1018.674614632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls") pod "maas-api-e2e-mt-f916d574-77bff56859-nnjr5" (UID: "e01c7126-4436-4844-babb-9810882174f2") : secret "maas-api-e2e-mt-f916d574-serving-cert" not found
Sep 16 12:18:48.537807 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.537784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7126-4436-4844-babb-9810882174f2-openshift-service-ca\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.539747 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.539728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-metrics-certs\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:48.547934 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:48.547913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfvl\" (UniqueName: \"kubernetes.io/projected/e01c7126-4436-4844-babb-9810882174f2-kube-api-access-6tfvl\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:49.042747 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:49.042707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:49.045235 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:49.045204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls\") pod \"maas-api-e2e-mt-f916d574-77bff56859-nnjr5\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:49.266342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:49.266308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:49.539399 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:49.539370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"]
Sep 16 12:18:49.540647 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:18:49.540621    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01c7126_4436_4844_babb_9810882174f2.slice/crio-bb07e20f08dbbcf851f10b260adf5dbb8171ebfff2a5080e11ea3aa4ef5d20f9 WatchSource:0}: Error finding container bb07e20f08dbbcf851f10b260adf5dbb8171ebfff2a5080e11ea3aa4ef5d20f9: Status 404 returned error can't find the container with id bb07e20f08dbbcf851f10b260adf5dbb8171ebfff2a5080e11ea3aa4ef5d20f9
Sep 16 12:18:49.542008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:49.541990    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:18:50.177266 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:50.177171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5" event={"ID":"e01c7126-4436-4844-babb-9810882174f2","Type":"ContainerStarted","Data":"1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077"}
Sep 16 12:18:50.177266 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:50.177241    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:18:50.177266 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:50.177267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5" event={"ID":"e01c7126-4436-4844-babb-9810882174f2","Type":"ContainerStarted","Data":"bb07e20f08dbbcf851f10b260adf5dbb8171ebfff2a5080e11ea3aa4ef5d20f9"}
Sep 16 12:18:50.197353 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:50.197287    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5" podStartSLOduration=1.820338779 podStartE2EDuration="2.197273541s" totalImagesPullingTime="376.934762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:18:50.195720463 +0000 UTC m=+1019.833045312" watchObservedRunningTime="2026-09-16 12:18:50.197273541 +0000 UTC m=+1019.834598378"
Sep 16 12:18:56.187723 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:18:56.187696    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:19:03.435337 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:03.435302    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"]
Sep 16 12:19:18.455184 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.455145    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v" podUID="a6f4c1a5-a084-4208-91c7-670ee6307503" containerName="main" containerID="cri-o://55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440" gracePeriod=45
Sep 16 12:19:18.719656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.719597    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:19:18.823758 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.823730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-tmp-dir\") pod \"a6f4c1a5-a084-4208-91c7-670ee6307503\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") "
Sep 16 12:19:18.823917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.823791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-dshm\") pod \"a6f4c1a5-a084-4208-91c7-670ee6307503\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") "
Sep 16 12:19:18.823917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.823834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f4c1a5-a084-4208-91c7-670ee6307503-tls-certs\") pod \"a6f4c1a5-a084-4208-91c7-670ee6307503\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") "
Sep 16 12:19:18.823917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.823860    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjfvf\" (UniqueName: \"kubernetes.io/projected/a6f4c1a5-a084-4208-91c7-670ee6307503-kube-api-access-bjfvf\") pod \"a6f4c1a5-a084-4208-91c7-670ee6307503\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") "
Sep 16 12:19:18.823917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.823888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-home\") pod \"a6f4c1a5-a084-4208-91c7-670ee6307503\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") "
Sep 16 12:19:18.824154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.823927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-model-cache\") pod \"a6f4c1a5-a084-4208-91c7-670ee6307503\" (UID: \"a6f4c1a5-a084-4208-91c7-670ee6307503\") "
Sep 16 12:19:18.824154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.824031    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6f4c1a5-a084-4208-91c7-670ee6307503" (UID: "a6f4c1a5-a084-4208-91c7-670ee6307503"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:18.824261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.824248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6f4c1a5-a084-4208-91c7-670ee6307503" (UID: "a6f4c1a5-a084-4208-91c7-670ee6307503"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:18.824261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.824243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-home" (OuterVolumeSpecName: "home") pod "a6f4c1a5-a084-4208-91c7-670ee6307503" (UID: "a6f4c1a5-a084-4208-91c7-670ee6307503"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:18.824387 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.824307    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:18.826255 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.826219    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f4c1a5-a084-4208-91c7-670ee6307503-kube-api-access-bjfvf" (OuterVolumeSpecName: "kube-api-access-bjfvf") pod "a6f4c1a5-a084-4208-91c7-670ee6307503" (UID: "a6f4c1a5-a084-4208-91c7-670ee6307503"). InnerVolumeSpecName "kube-api-access-bjfvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:18.826364 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.826251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-dshm" (OuterVolumeSpecName: "dshm") pod "a6f4c1a5-a084-4208-91c7-670ee6307503" (UID: "a6f4c1a5-a084-4208-91c7-670ee6307503"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:18.826364 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.826254    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f4c1a5-a084-4208-91c7-670ee6307503-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6f4c1a5-a084-4208-91c7-670ee6307503" (UID: "a6f4c1a5-a084-4208-91c7-670ee6307503"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:19:18.924751 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.924724    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f4c1a5-a084-4208-91c7-670ee6307503-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:18.924751 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.924747    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfvf\" (UniqueName: \"kubernetes.io/projected/a6f4c1a5-a084-4208-91c7-670ee6307503-kube-api-access-bjfvf\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:18.924887 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.924757    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:18.924887 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.924766    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:18.924887 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:18.924776    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f4c1a5-a084-4208-91c7-670ee6307503-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:19.344821 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.344783    2578 generic.go:356] "Generic (PLEG): container finished" podID="a6f4c1a5-a084-4208-91c7-670ee6307503" containerID="55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440" exitCode=0
Sep 16 12:19:19.345011 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.344828    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"
Sep 16 12:19:19.345011 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.344868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v" event={"ID":"a6f4c1a5-a084-4208-91c7-670ee6307503","Type":"ContainerDied","Data":"55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440"}
Sep 16 12:19:19.345011 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.344917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v" event={"ID":"a6f4c1a5-a084-4208-91c7-670ee6307503","Type":"ContainerDied","Data":"4c6fb889500d211740cb2e75d92ff962e8ee1917c50eb40fc5c10d3a8a4cda69"}
Sep 16 12:19:19.345011 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.344949    2578 scope.go:117] "RemoveContainer" containerID="55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440"
Sep 16 12:19:19.354607 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.354586    2578 scope.go:117] "RemoveContainer" containerID="55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440"
Sep 16 12:19:19.354893 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:19:19.354864    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440\": container with ID starting with 55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440 not found: ID does not exist" containerID="55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440"
Sep 16 12:19:19.354977 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.354900    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440"} err="failed to get container status \"55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440\": rpc error: code = NotFound desc = could not find container \"55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440\": container with ID starting with 55ca36fa729b09f9ac476b400e29ec6da5804d0a022f30553dc79b37cd11a440 not found: ID does not exist"
Sep 16 12:19:19.361189 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.361163    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"]
Sep 16 12:19:19.364555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:19.364535    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-7810a5/explicit-model-7810a5-kserve-cf44c9c5c-sdk7v"]
Sep 16 12:19:20.885026 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:20.884989    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f4c1a5-a084-4208-91c7-670ee6307503" path="/var/lib/kubelet/pods/a6f4c1a5-a084-4208-91c7-670ee6307503/volumes"
Sep 16 12:19:42.137946 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:42.137863    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"]
Sep 16 12:19:47.980225 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:47.980190    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"]
Sep 16 12:19:47.982803 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:47.980434    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5" podUID="e01c7126-4436-4844-babb-9810882174f2" containerName="maas-api" containerID="cri-o://1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077" gracePeriod=30
Sep 16 12:19:48.231263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.231204    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:19:48.391825 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.391793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-metrics-certs\") pod \"e01c7126-4436-4844-babb-9810882174f2\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") "
Sep 16 12:19:48.391825 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.391831    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls\") pod \"e01c7126-4436-4844-babb-9810882174f2\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") "
Sep 16 12:19:48.392088 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.391917    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tfvl\" (UniqueName: \"kubernetes.io/projected/e01c7126-4436-4844-babb-9810882174f2-kube-api-access-6tfvl\") pod \"e01c7126-4436-4844-babb-9810882174f2\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") "
Sep 16 12:19:48.392088 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.391943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7126-4436-4844-babb-9810882174f2-openshift-service-ca\") pod \"e01c7126-4436-4844-babb-9810882174f2\" (UID: \"e01c7126-4436-4844-babb-9810882174f2\") "
Sep 16 12:19:48.392374 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.392339    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01c7126-4436-4844-babb-9810882174f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e01c7126-4436-4844-babb-9810882174f2" (UID: "e01c7126-4436-4844-babb-9810882174f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:19:48.394247 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.394221    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e01c7126-4436-4844-babb-9810882174f2" (UID: "e01c7126-4436-4844-babb-9810882174f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:19:48.394247 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.394237    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01c7126-4436-4844-babb-9810882174f2-kube-api-access-6tfvl" (OuterVolumeSpecName: "kube-api-access-6tfvl") pod "e01c7126-4436-4844-babb-9810882174f2" (UID: "e01c7126-4436-4844-babb-9810882174f2"). InnerVolumeSpecName "kube-api-access-6tfvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:48.394425 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.394246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e01c7126-4436-4844-babb-9810882174f2" (UID: "e01c7126-4436-4844-babb-9810882174f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:19:48.493413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.493335    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:48.493413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.493370    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e01c7126-4436-4844-babb-9810882174f2-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:48.493413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.493381    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfvl\" (UniqueName: \"kubernetes.io/projected/e01c7126-4436-4844-babb-9810882174f2-kube-api-access-6tfvl\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:48.493413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.493391    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7126-4436-4844-babb-9810882174f2-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:48.506227 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.506193    2578 generic.go:356] "Generic (PLEG): container finished" podID="e01c7126-4436-4844-babb-9810882174f2" containerID="1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077" exitCode=0
Sep 16 12:19:48.506331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.506278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5" event={"ID":"e01c7126-4436-4844-babb-9810882174f2","Type":"ContainerDied","Data":"1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077"}
Sep 16 12:19:48.506331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.506304    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"
Sep 16 12:19:48.506331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.506324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5" event={"ID":"e01c7126-4436-4844-babb-9810882174f2","Type":"ContainerDied","Data":"bb07e20f08dbbcf851f10b260adf5dbb8171ebfff2a5080e11ea3aa4ef5d20f9"}
Sep 16 12:19:48.506458 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.506344    2578 scope.go:117] "RemoveContainer" containerID="1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077"
Sep 16 12:19:48.517278 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.517259    2578 scope.go:117] "RemoveContainer" containerID="1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077"
Sep 16 12:19:48.517624 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:19:48.517600    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077\": container with ID starting with 1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077 not found: ID does not exist" containerID="1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077"
Sep 16 12:19:48.517708 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.517634    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077"} err="failed to get container status \"1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077\": rpc error: code = NotFound desc = could not find container \"1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077\": container with ID starting with 1aad7920508e58cd2a205d2a00a22573bfdd0019ddac818c2afb26b7e32ce077 not found: ID does not exist"
Sep 16 12:19:48.529171 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.529149    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"]
Sep 16 12:19:48.532992 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.532972    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f916d574-77bff56859-nnjr5"]
Sep 16 12:19:48.885263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:48.885185    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e01c7126-4436-4844-babb-9810882174f2" path="/var/lib/kubelet/pods/e01c7126-4436-4844-babb-9810882174f2/volumes"
Sep 16 12:19:57.158578 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.158507    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8" podUID="d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" containerName="main" containerID="cri-o://fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381" gracePeriod=45
Sep 16 12:19:57.401868 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.401836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:19:57.484190 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484158    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tmp-dir\") pod \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") "
Sep 16 12:19:57.484367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-model-cache\") pod \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") "
Sep 16 12:19:57.484367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7f9x\" (UniqueName: \"kubernetes.io/projected/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-kube-api-access-h7f9x\") pod \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") "
Sep 16 12:19:57.484367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-home\") pod \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") "
Sep 16 12:19:57.484367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484306    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-dshm\") pod \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") "
Sep 16 12:19:57.484367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484360    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tls-certs\") pod \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\" (UID: \"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f\") "
Sep 16 12:19:57.484632 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" (UID: "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:57.484632 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" (UID: "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:57.484632 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-home" (OuterVolumeSpecName: "home") pod "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" (UID: "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:57.484760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484714    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:57.484760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484727    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:57.484760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.484736    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:57.486723 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.486693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-dshm" (OuterVolumeSpecName: "dshm") pod "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" (UID: "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:57.486828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.486752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" (UID: "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:19:57.486828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.486806    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-kube-api-access-h7f9x" (OuterVolumeSpecName: "kube-api-access-h7f9x") pod "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" (UID: "d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f"). InnerVolumeSpecName "kube-api-access-h7f9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:57.555827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.555792    2578 generic.go:356] "Generic (PLEG): container finished" podID="d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" containerID="fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381" exitCode=0
Sep 16 12:19:57.555963 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.555841    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"
Sep 16 12:19:57.555963 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.555844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8" event={"ID":"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f","Type":"ContainerDied","Data":"fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381"}
Sep 16 12:19:57.555963 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.555891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8" event={"ID":"d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f","Type":"ContainerDied","Data":"238d7209f80f71ec82f50a9bfa8ec895dd651024e027b83796ee9d3640678971"}
Sep 16 12:19:57.555963 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.555908    2578 scope.go:117] "RemoveContainer" containerID="fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381"
Sep 16 12:19:57.565460 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.565443    2578 scope.go:117] "RemoveContainer" containerID="fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381"
Sep 16 12:19:57.565741 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:19:57.565723    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381\": container with ID starting with fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381 not found: ID does not exist" containerID="fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381"
Sep 16 12:19:57.565793 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.565749    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381"} err="failed to get container status \"fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381\": rpc error: code = NotFound desc = could not find container \"fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381\": container with ID starting with fb24e10a5cc8c996a126af9add10760d17fa8ffac4068786d68dee13e0137381 not found: ID does not exist"
Sep 16 12:19:57.578242 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.578218    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"]
Sep 16 12:19:57.584608 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.584588    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-641983/test-model-641983-kserve-766d6b5fc9-zr8k8"]
Sep 16 12:19:57.585598 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.585576    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7f9x\" (UniqueName: \"kubernetes.io/projected/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-kube-api-access-h7f9x\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:57.585598 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.585598    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:57.585735 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:57.585607    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:19:58.884074 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:19:58.884042    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" path="/var/lib/kubelet/pods/d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f/volumes"
Sep 16 12:20:07.206399 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:07.206359    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"]
Sep 16 12:20:18.987046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:18.987014    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"]
Sep 16 12:20:18.987565 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:18.987217    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w" podUID="58a7cde5-d85c-4813-aa8c-1aae01f4e002" containerName="maas-api" containerID="cri-o://cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8" gracePeriod=30
Sep 16 12:20:19.242360 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.242298    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:20:19.394877 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.394844    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbrvq\" (UniqueName: \"kubernetes.io/projected/58a7cde5-d85c-4813-aa8c-1aae01f4e002-kube-api-access-gbrvq\") pod \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") "
Sep 16 12:20:19.395045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.394913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a7cde5-d85c-4813-aa8c-1aae01f4e002-openshift-service-ca\") pod \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") "
Sep 16 12:20:19.395045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.394948    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls\") pod \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") "
Sep 16 12:20:19.395045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.394968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-metrics-certs\") pod \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\" (UID: \"58a7cde5-d85c-4813-aa8c-1aae01f4e002\") "
Sep 16 12:20:19.395316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.395287    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a7cde5-d85c-4813-aa8c-1aae01f4e002-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58a7cde5-d85c-4813-aa8c-1aae01f4e002" (UID: "58a7cde5-d85c-4813-aa8c-1aae01f4e002"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:19.397221 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.397198    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "58a7cde5-d85c-4813-aa8c-1aae01f4e002" (UID: "58a7cde5-d85c-4813-aa8c-1aae01f4e002"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:19.397298 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.397238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58a7cde5-d85c-4813-aa8c-1aae01f4e002" (UID: "58a7cde5-d85c-4813-aa8c-1aae01f4e002"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:19.397367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.397350    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a7cde5-d85c-4813-aa8c-1aae01f4e002-kube-api-access-gbrvq" (OuterVolumeSpecName: "kube-api-access-gbrvq") pod "58a7cde5-d85c-4813-aa8c-1aae01f4e002" (UID: "58a7cde5-d85c-4813-aa8c-1aae01f4e002"). InnerVolumeSpecName "kube-api-access-gbrvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:19.496348 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.496274    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a7cde5-d85c-4813-aa8c-1aae01f4e002-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:19.496348 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.496299    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:19.496348 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.496308    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a7cde5-d85c-4813-aa8c-1aae01f4e002-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:19.496348 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.496317    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrvq\" (UniqueName: \"kubernetes.io/projected/58a7cde5-d85c-4813-aa8c-1aae01f4e002-kube-api-access-gbrvq\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:19.689989 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.689958    2578 generic.go:356] "Generic (PLEG): container finished" podID="58a7cde5-d85c-4813-aa8c-1aae01f4e002" containerID="cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8" exitCode=0
Sep 16 12:20:19.690160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.690001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w" event={"ID":"58a7cde5-d85c-4813-aa8c-1aae01f4e002","Type":"ContainerDied","Data":"cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8"}
Sep 16 12:20:19.690160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.690006    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"
Sep 16 12:20:19.690160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.690042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w" event={"ID":"58a7cde5-d85c-4813-aa8c-1aae01f4e002","Type":"ContainerDied","Data":"40b072cbee47e62b2ca803e54a687a2cf5c086cb157608823f7b396828a43364"}
Sep 16 12:20:19.690160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.690065    2578 scope.go:117] "RemoveContainer" containerID="cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8"
Sep 16 12:20:19.700431 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.700410    2578 scope.go:117] "RemoveContainer" containerID="cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8"
Sep 16 12:20:19.700751 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:20:19.700730    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8\": container with ID starting with cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8 not found: ID does not exist" containerID="cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8"
Sep 16 12:20:19.700849 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.700758    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8"} err="failed to get container status \"cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8\": rpc error: code = NotFound desc = could not find container \"cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8\": container with ID starting with cfccc4d444c5fb756600cece547156ba25951621c1de6feaeaf8f6795d35efd8 not found: ID does not exist"
Sep 16 12:20:19.720379 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.720355    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"]
Sep 16 12:20:19.728657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:19.728636    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-641983-7bc847cb5b-j4w7w"]
Sep 16 12:20:20.885918 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:20.885874    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a7cde5-d85c-4813-aa8c-1aae01f4e002" path="/var/lib/kubelet/pods/58a7cde5-d85c-4813-aa8c-1aae01f4e002/volumes"
Sep 16 12:20:22.237292 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.237256    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z" podUID="82faedf9-2bc6-47f4-a298-bb056e2fa89a" containerName="main" containerID="cri-o://2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea" gracePeriod=45
Sep 16 12:20:22.485167 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.485140    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:20:22.624840 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.624756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-dshm\") pod \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") "
Sep 16 12:20:22.624840 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.624814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tmp-dir\") pod \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") "
Sep 16 12:20:22.625082 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.624889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-model-cache\") pod \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") "
Sep 16 12:20:22.625082 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.624943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-home\") pod \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") "
Sep 16 12:20:22.625082 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.625003    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s77w\" (UniqueName: \"kubernetes.io/projected/82faedf9-2bc6-47f4-a298-bb056e2fa89a-kube-api-access-4s77w\") pod \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") "
Sep 16 12:20:22.625082 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.625026    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tls-certs\") pod \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\" (UID: \"82faedf9-2bc6-47f4-a298-bb056e2fa89a\") "
Sep 16 12:20:22.625274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.625110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82faedf9-2bc6-47f4-a298-bb056e2fa89a" (UID: "82faedf9-2bc6-47f4-a298-bb056e2fa89a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:22.625274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.625130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-model-cache" (OuterVolumeSpecName: "model-cache") pod "82faedf9-2bc6-47f4-a298-bb056e2fa89a" (UID: "82faedf9-2bc6-47f4-a298-bb056e2fa89a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:22.625274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.625252    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-home" (OuterVolumeSpecName: "home") pod "82faedf9-2bc6-47f4-a298-bb056e2fa89a" (UID: "82faedf9-2bc6-47f4-a298-bb056e2fa89a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:22.625413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.625368    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:22.625413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.625388    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:22.625499 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.625422    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:22.627277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.627210    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-dshm" (OuterVolumeSpecName: "dshm") pod "82faedf9-2bc6-47f4-a298-bb056e2fa89a" (UID: "82faedf9-2bc6-47f4-a298-bb056e2fa89a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:22.627277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.627262    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82faedf9-2bc6-47f4-a298-bb056e2fa89a-kube-api-access-4s77w" (OuterVolumeSpecName: "kube-api-access-4s77w") pod "82faedf9-2bc6-47f4-a298-bb056e2fa89a" (UID: "82faedf9-2bc6-47f4-a298-bb056e2fa89a"). InnerVolumeSpecName "kube-api-access-4s77w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:22.627448 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.627364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82faedf9-2bc6-47f4-a298-bb056e2fa89a" (UID: "82faedf9-2bc6-47f4-a298-bb056e2fa89a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:22.709858 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.709830    2578 generic.go:356] "Generic (PLEG): container finished" podID="82faedf9-2bc6-47f4-a298-bb056e2fa89a" containerID="2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea" exitCode=0
Sep 16 12:20:22.709986 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.709877    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"
Sep 16 12:20:22.709986 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.709923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z" event={"ID":"82faedf9-2bc6-47f4-a298-bb056e2fa89a","Type":"ContainerDied","Data":"2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea"}
Sep 16 12:20:22.709986 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.709965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z" event={"ID":"82faedf9-2bc6-47f4-a298-bb056e2fa89a","Type":"ContainerDied","Data":"9a117aa37699f92b3368f24943b48cd81c8767800c5008d74d1e8addc9d9e8a5"}
Sep 16 12:20:22.709986 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.709982    2578 scope.go:117] "RemoveContainer" containerID="2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea"
Sep 16 12:20:22.719697 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.719682    2578 scope.go:117] "RemoveContainer" containerID="2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea"
Sep 16 12:20:22.719962 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:20:22.719946    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea\": container with ID starting with 2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea not found: ID does not exist" containerID="2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea"
Sep 16 12:20:22.720007 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.719972    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea"} err="failed to get container status \"2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea\": rpc error: code = NotFound desc = could not find container \"2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea\": container with ID starting with 2a6bf45e2aa3c61f1003f9883e021b2f727ad0e13d6cd711c595d1fe4e30a9ea not found: ID does not exist"
Sep 16 12:20:22.725966 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.725949    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s77w\" (UniqueName: \"kubernetes.io/projected/82faedf9-2bc6-47f4-a298-bb056e2fa89a-kube-api-access-4s77w\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:22.726033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.725968    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82faedf9-2bc6-47f4-a298-bb056e2fa89a-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:22.726033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.725976    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82faedf9-2bc6-47f4-a298-bb056e2fa89a-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:22.731811 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.731791    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"]
Sep 16 12:20:22.735547 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.735527    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-08170a/test-model-08170a-kserve-b67c47ddc-gls2z"]
Sep 16 12:20:22.884797 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:22.884727    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82faedf9-2bc6-47f4-a298-bb056e2fa89a" path="/var/lib/kubelet/pods/82faedf9-2bc6-47f4-a298-bb056e2fa89a/volumes"
Sep 16 12:20:34.022445 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.022405    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"]
Sep 16 12:20:34.022993 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.022970    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" containerName="main"
Sep 16 12:20:34.023040 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.022998    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" containerName="main"
Sep 16 12:20:34.023040 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023015    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01c7126-4436-4844-babb-9810882174f2" containerName="maas-api"
Sep 16 12:20:34.023040 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023024    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01c7126-4436-4844-babb-9810882174f2" containerName="maas-api"
Sep 16 12:20:34.023137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023058    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82faedf9-2bc6-47f4-a298-bb056e2fa89a" containerName="main"
Sep 16 12:20:34.023137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023066    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="82faedf9-2bc6-47f4-a298-bb056e2fa89a" containerName="main"
Sep 16 12:20:34.023137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023081    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f4c1a5-a084-4208-91c7-670ee6307503" containerName="main"
Sep 16 12:20:34.023137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023089    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f4c1a5-a084-4208-91c7-670ee6307503" containerName="main"
Sep 16 12:20:34.023137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023102    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a7cde5-d85c-4813-aa8c-1aae01f4e002" containerName="maas-api"
Sep 16 12:20:34.023137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023110    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a7cde5-d85c-4813-aa8c-1aae01f4e002" containerName="maas-api"
Sep 16 12:20:34.023316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023206    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95f44c5-4f1e-4e3a-b862-fc9c5cc9034f" containerName="main"
Sep 16 12:20:34.023316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023223    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="82faedf9-2bc6-47f4-a298-bb056e2fa89a" containerName="main"
Sep 16 12:20:34.023316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023233    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f4c1a5-a084-4208-91c7-670ee6307503" containerName="main"
Sep 16 12:20:34.023316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023242    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a7cde5-d85c-4813-aa8c-1aae01f4e002" containerName="maas-api"
Sep 16 12:20:34.023316 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.023252    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01c7126-4436-4844-babb-9810882174f2" containerName="maas-api"
Sep 16 12:20:34.026886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.026852    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.029582 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.029558    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6f8610-openshift-default-dockercfg-qqnc2\""
Sep 16 12:20:34.029726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.029558    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:20:34.038552 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.038494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"]
Sep 16 12:20:34.133964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.133927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-socket\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.133964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.133965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-token\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.134173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.133987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-credential-socket\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.134173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.134009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-podinfo\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.134173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.134043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istiod-ca-cert\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.134173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.134093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-data\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.134173 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.134127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7nw\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-kube-api-access-kk7nw\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.134361 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.134197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-certs\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.134361 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.134214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-envoy\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-certs\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-envoy\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-socket\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-token\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-credential-socket\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-podinfo\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istiod-ca-cert\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-data\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7nw\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-kube-api-access-kk7nw\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-socket\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235834 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-certs\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.235834 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-credential-socket\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.236125 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.236109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istiod-ca-cert\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.236166 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.235985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-data\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.237715 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.237693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-envoy\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.237964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.237944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-podinfo\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.243317 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.243294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-token\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.243401 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.243370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7nw\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-kube-api-access-kk7nw\") pod \"e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") " pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.338552 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.338459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:34.614150 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.614122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"]
Sep 16 12:20:34.615224 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:20:34.615192    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6fc64a_c1a9_41dd_aa41_4f08230fd589.slice/crio-af134835fb849ed238fc63449525ad4b58129f0321b42c432955d14df2b1584a WatchSource:0}: Error finding container af134835fb849ed238fc63449525ad4b58129f0321b42c432955d14df2b1584a: Status 404 returned error can't find the container with id af134835fb849ed238fc63449525ad4b58129f0321b42c432955d14df2b1584a
Sep 16 12:20:34.617528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.617482    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:20:34.617641 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.617576    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:20:34.617641 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.617607    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:20:34.781123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.781082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" event={"ID":"ba6fc64a-c1a9-41dd-aa41-4f08230fd589","Type":"ContainerStarted","Data":"5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf"}
Sep 16 12:20:34.781281 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.781139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" event={"ID":"ba6fc64a-c1a9-41dd-aa41-4f08230fd589","Type":"ContainerStarted","Data":"af134835fb849ed238fc63449525ad4b58129f0321b42c432955d14df2b1584a"}
Sep 16 12:20:34.801216 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:34.801166    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" podStartSLOduration=0.801149597 podStartE2EDuration="801.149597ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:34.799455088 +0000 UTC m=+1124.436779924" watchObservedRunningTime="2026-09-16 12:20:34.801149597 +0000 UTC m=+1124.438474435"
Sep 16 12:20:35.339220 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:35.339190    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:35.344857 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:35.344830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:35.786140 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:35.786111    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:35.787157 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:35.787137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:20:39.094695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.094660    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"]
Sep 16 12:20:39.095145 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.094912    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl" podUID="659f3f12-e436-4b61-a5fd-9b36cfa7db7a" containerName="maas-api" containerID="cri-o://6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f" gracePeriod=30
Sep 16 12:20:39.124663 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:20:39.124635    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659f3f12_e436_4b61_a5fd_9b36cfa7db7a.slice/crio-conmon-6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 12:20:39.350182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.350117    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:20:39.360325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.360301    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"]
Sep 16 12:20:39.360700 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.360687    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659f3f12-e436-4b61-a5fd-9b36cfa7db7a" containerName="maas-api"
Sep 16 12:20:39.360746 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.360702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="659f3f12-e436-4b61-a5fd-9b36cfa7db7a" containerName="maas-api"
Sep 16 12:20:39.360796 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.360772    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="659f3f12-e436-4b61-a5fd-9b36cfa7db7a" containerName="maas-api"
Sep 16 12:20:39.365407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.365384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.367890 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.367869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-6f8610-serving-cert\""
Sep 16 12:20:39.376651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.376625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"]
Sep 16 12:20:39.483244 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6czw\" (UniqueName: \"kubernetes.io/projected/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-kube-api-access-f6czw\") pod \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") "
Sep 16 12:20:39.483417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483257    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-metrics-certs\") pod \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") "
Sep 16 12:20:39.483417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483293    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls\") pod \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") "
Sep 16 12:20:39.483417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-openshift-service-ca\") pod \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\" (UID: \"659f3f12-e436-4b61-a5fd-9b36cfa7db7a\") "
Sep 16 12:20:39.483642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.483642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.483642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpb2l\" (UniqueName: \"kubernetes.io/projected/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-kube-api-access-mpb2l\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.483777 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.483830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.483794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "659f3f12-e436-4b61-a5fd-9b36cfa7db7a" (UID: "659f3f12-e436-4b61-a5fd-9b36cfa7db7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:39.485632 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.485608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "659f3f12-e436-4b61-a5fd-9b36cfa7db7a" (UID: "659f3f12-e436-4b61-a5fd-9b36cfa7db7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:39.486115 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.486087    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "659f3f12-e436-4b61-a5fd-9b36cfa7db7a" (UID: "659f3f12-e436-4b61-a5fd-9b36cfa7db7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:20:39.486200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.486110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-kube-api-access-f6czw" (OuterVolumeSpecName: "kube-api-access-f6czw") pod "659f3f12-e436-4b61-a5fd-9b36cfa7db7a" (UID: "659f3f12-e436-4b61-a5fd-9b36cfa7db7a"). InnerVolumeSpecName "kube-api-access-f6czw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:39.584446 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.584408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.584651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.584457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.584651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.584483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpb2l\" (UniqueName: \"kubernetes.io/projected/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-kube-api-access-mpb2l\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.584651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.584564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.584651 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:20:39.584581    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-serving-cert: secret "maas-api-e2e-shared-b-w3-6f8610-serving-cert" not found
Sep 16 12:20:39.584651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.584600    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6czw\" (UniqueName: \"kubernetes.io/projected/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-kube-api-access-f6czw\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:39.584651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.584611    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:39.584651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.584620    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:39.584651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.584629    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f3f12-e436-4b61-a5fd-9b36cfa7db7a-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:20:39.584956 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:20:39.584658    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls podName:5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678 nodeName:}" failed. No retries permitted until 2026-09-16 12:20:40.084636902 +0000 UTC m=+1129.721961733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls") pod "maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb" (UID: "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678") : secret "maas-api-e2e-shared-b-w3-6f8610-serving-cert" not found
Sep 16 12:20:39.585254 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.585236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.587051 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.587027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.595172 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.595154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpb2l\" (UniqueName: \"kubernetes.io/projected/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-kube-api-access-mpb2l\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:39.814767 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.814733    2578 generic.go:356] "Generic (PLEG): container finished" podID="659f3f12-e436-4b61-a5fd-9b36cfa7db7a" containerID="6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f" exitCode=0
Sep 16 12:20:39.814937 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.814784    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"
Sep 16 12:20:39.814937 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.814806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl" event={"ID":"659f3f12-e436-4b61-a5fd-9b36cfa7db7a","Type":"ContainerDied","Data":"6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f"}
Sep 16 12:20:39.814937 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.814858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl" event={"ID":"659f3f12-e436-4b61-a5fd-9b36cfa7db7a","Type":"ContainerDied","Data":"7c69eaebd2e32389b2a64f8cfd431faf4b1240344c2dadd4eca0175fc0c7c961"}
Sep 16 12:20:39.814937 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.814884    2578 scope.go:117] "RemoveContainer" containerID="6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f"
Sep 16 12:20:39.826099 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.826080    2578 scope.go:117] "RemoveContainer" containerID="6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f"
Sep 16 12:20:39.826398 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:20:39.826378    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f\": container with ID starting with 6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f not found: ID does not exist" containerID="6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f"
Sep 16 12:20:39.826452 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.826407    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f"} err="failed to get container status \"6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f\": rpc error: code = NotFound desc = could not find container \"6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f\": container with ID starting with 6e06ea16b37ce5127199ec41030114895f618293917e9c76ccb2d7eda982254f not found: ID does not exist"
Sep 16 12:20:39.839229 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.839197    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"]
Sep 16 12:20:39.843562 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:39.843540    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08170a-5877557b75-82qrl"]
Sep 16 12:20:40.089116 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:40.089029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:40.091339 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:40.091321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:40.277670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:40.277638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:40.554957 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:40.554932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"]
Sep 16 12:20:40.556483 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:20:40.556449    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6504b0_d41b_4d9a_ac8e_a3bcfb5dd678.slice/crio-763120e01cf8b3d2394a7f3a9042455ddb84e6e7fe1058c88cd767ac65eed4f7 WatchSource:0}: Error finding container 763120e01cf8b3d2394a7f3a9042455ddb84e6e7fe1058c88cd767ac65eed4f7: Status 404 returned error can't find the container with id 763120e01cf8b3d2394a7f3a9042455ddb84e6e7fe1058c88cd767ac65eed4f7
Sep 16 12:20:40.822263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:40.822170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb" event={"ID":"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678","Type":"ContainerStarted","Data":"763120e01cf8b3d2394a7f3a9042455ddb84e6e7fe1058c88cd767ac65eed4f7"}
Sep 16 12:20:40.888436 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:40.888401    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="659f3f12-e436-4b61-a5fd-9b36cfa7db7a" path="/var/lib/kubelet/pods/659f3f12-e436-4b61-a5fd-9b36cfa7db7a/volumes"
Sep 16 12:20:41.828876 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:41.828843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb" event={"ID":"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678","Type":"ContainerStarted","Data":"5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a"}
Sep 16 12:20:41.829392 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:41.828970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:41.847077 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:41.847033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb" podStartSLOduration=2.415579123 podStartE2EDuration="2.847018269s" totalImagesPullingTime="431.439146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:41.845887575 +0000 UTC m=+1131.483212443" watchObservedRunningTime="2026-09-16 12:20:41.847018269 +0000 UTC m=+1131.484343135"
Sep 16 12:20:46.825756 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.825721    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"]
Sep 16 12:20:46.829834 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.829809    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.832797 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.832775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-db42a6-openshift-default-dockercfg-pz6ck\""
Sep 16 12:20:46.839317 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.839290    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"]
Sep 16 12:20:46.849459 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.849397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-data\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.849658 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.849631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istiod-ca-cert\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.849962 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.849941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-envoy\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.850098 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.850080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl2d\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-kube-api-access-4pl2d\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.850247 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.850224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-certs\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.850468 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.850271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-credential-socket\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.850468 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.850328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-socket\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.850468 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.850356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-token\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.850656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.850467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-podinfo\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.951846 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.951811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-socket\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.951846 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.951847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-token\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.951878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-podinfo\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.951904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-data\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.951943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istiod-ca-cert\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-envoy\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl2d\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-kube-api-access-4pl2d\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952277 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-socket\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952387 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-data\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952446 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-certs\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952446 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-credential-socket\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952750 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istiod-ca-cert\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952835 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-credential-socket\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.952939 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.952916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-certs\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.954436 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.954415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-envoy\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.954567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.954547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-podinfo\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.960216 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.960194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-token\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:46.960501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:46.960482    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl2d\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-kube-api-access-4pl2d\") pod \"e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") " pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:47.141778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.141700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:47.416508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.416473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"]
Sep 16 12:20:47.417660 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:20:47.417634    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63859a7d_a1da_45dd_abc0_95bdd0023b5c.slice/crio-9537e4db54c54c651419060dc6a909a5b8ac3cb1863ec72b06a2bfc747e84510 WatchSource:0}: Error finding container 9537e4db54c54c651419060dc6a909a5b8ac3cb1863ec72b06a2bfc747e84510: Status 404 returned error can't find the container with id 9537e4db54c54c651419060dc6a909a5b8ac3cb1863ec72b06a2bfc747e84510
Sep 16 12:20:47.419934 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.419903    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:20:47.420020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.419965    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:20:47.420020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.420003    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:20:47.839884 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.839854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:20:47.864045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.864002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" event={"ID":"63859a7d-a1da-45dd-abc0-95bdd0023b5c","Type":"ContainerStarted","Data":"4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb"}
Sep 16 12:20:47.864212 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.864065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" event={"ID":"63859a7d-a1da-45dd-abc0-95bdd0023b5c","Type":"ContainerStarted","Data":"9537e4db54c54c651419060dc6a909a5b8ac3cb1863ec72b06a2bfc747e84510"}
Sep 16 12:20:47.893018 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:47.892957    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podStartSLOduration=1.892934595 podStartE2EDuration="1.892934595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:47.889540723 +0000 UTC m=+1137.526865562" watchObservedRunningTime="2026-09-16 12:20:47.892934595 +0000 UTC m=+1137.530259434"
Sep 16 12:20:48.142141 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:48.142058    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:48.147441 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:48.147418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:48.870074 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:48.870038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:48.870874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:48.870857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:20:55.036123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.036089    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"]
Sep 16 12:20:55.042035 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.042002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.046332 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.046303    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c26b8c\"/\"default-dockercfg-8tggf\""
Sep 16 12:20:55.046594 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.046571    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c26b8c\"/\"kube-root-ca.crt\""
Sep 16 12:20:55.046809 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.046787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c26b8c\"/\"openshift-service-ca.crt\""
Sep 16 12:20:55.047016 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.046993    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c26b8c\"/\"noten-model-c26b8c-kserve-self-signed-certs\""
Sep 16 12:20:55.052423 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.052397    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"]
Sep 16 12:20:55.138068 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.138021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-dshm\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.138272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.138076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-model-cache\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.138272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.138109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-tmp-dir\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.138272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.138152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-home\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.138272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.138193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb62fdb3-3971-46a1-ba49-2d6851843606-tls-certs\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.138272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.138219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ztf\" (UniqueName: \"kubernetes.io/projected/eb62fdb3-3971-46a1-ba49-2d6851843606-kube-api-access-j7ztf\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.239121 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.239073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-dshm\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.239329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.239129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-model-cache\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.239329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.239166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-tmp-dir\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.239329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.239210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-home\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.239329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.239249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb62fdb3-3971-46a1-ba49-2d6851843606-tls-certs\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.239329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.239275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ztf\" (UniqueName: \"kubernetes.io/projected/eb62fdb3-3971-46a1-ba49-2d6851843606-kube-api-access-j7ztf\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.240358 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.240330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-tmp-dir\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.241006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.240937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-model-cache\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.241591 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.241534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-home\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.242913 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.242894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-dshm\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.243586 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.243534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb62fdb3-3971-46a1-ba49-2d6851843606-tls-certs\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.247897 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.247872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ztf\" (UniqueName: \"kubernetes.io/projected/eb62fdb3-3971-46a1-ba49-2d6851843606-kube-api-access-j7ztf\") pod \"noten-model-c26b8c-kserve-bb6cf5467-rd22p\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") " pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.363597 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.363497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:20:55.636096 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.636004    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"]
Sep 16 12:20:55.646359 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:20:55.646319    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb62fdb3_3971_46a1_ba49_2d6851843606.slice/crio-5ffe745e395489ef72be70e026e3b40f5d7f12402032c06cd0bc10a1005a319f WatchSource:0}: Error finding container 5ffe745e395489ef72be70e026e3b40f5d7f12402032c06cd0bc10a1005a319f: Status 404 returned error can't find the container with id 5ffe745e395489ef72be70e026e3b40f5d7f12402032c06cd0bc10a1005a319f
Sep 16 12:20:55.907008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.906913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p" event={"ID":"eb62fdb3-3971-46a1-ba49-2d6851843606","Type":"ContainerStarted","Data":"efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b"}
Sep 16 12:20:55.907008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.906952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p" event={"ID":"eb62fdb3-3971-46a1-ba49-2d6851843606","Type":"ContainerStarted","Data":"5ffe745e395489ef72be70e026e3b40f5d7f12402032c06cd0bc10a1005a319f"}
Sep 16 12:20:55.925175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:55.925120    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p" podStartSLOduration=0.925104394 podStartE2EDuration="925.104394ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:55.924032561 +0000 UTC m=+1145.561357402" watchObservedRunningTime="2026-09-16 12:20:55.925104394 +0000 UTC m=+1145.562429232"
Sep 16 12:20:56.601742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:56.601705    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-e80130"]
Sep 16 12:20:56.605303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:56.605281    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-e80130"
Sep 16 12:20:56.608072 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:56.608049    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zpzr9\""
Sep 16 12:20:56.610774 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:56.610749    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-e80130"]
Sep 16 12:20:56.660135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:56.660095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5fwr\" (UniqueName: \"kubernetes.io/projected/82298439-954f-4a86-901e-b5798c5958a6-kube-api-access-g5fwr\") pod \"test-curl-7481-e80130\" (UID: \"82298439-954f-4a86-901e-b5798c5958a6\") " pod="opendatahub/test-curl-7481-e80130"
Sep 16 12:20:56.761127 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:56.761092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5fwr\" (UniqueName: \"kubernetes.io/projected/82298439-954f-4a86-901e-b5798c5958a6-kube-api-access-g5fwr\") pod \"test-curl-7481-e80130\" (UID: \"82298439-954f-4a86-901e-b5798c5958a6\") " pod="opendatahub/test-curl-7481-e80130"
Sep 16 12:20:56.769729 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:56.769694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5fwr\" (UniqueName: \"kubernetes.io/projected/82298439-954f-4a86-901e-b5798c5958a6-kube-api-access-g5fwr\") pod \"test-curl-7481-e80130\" (UID: \"82298439-954f-4a86-901e-b5798c5958a6\") " pod="opendatahub/test-curl-7481-e80130"
Sep 16 12:20:56.916656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:56.916567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-e80130"
Sep 16 12:20:57.206886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:57.206859    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-e80130"]
Sep 16 12:20:57.207914 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:20:57.207892    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82298439_954f_4a86_901e_b5798c5958a6.slice/crio-c64e86def19e3b69629142ddf8fa2b37a772d20ac9862ad8dbfed65559d0aca2 WatchSource:0}: Error finding container c64e86def19e3b69629142ddf8fa2b37a772d20ac9862ad8dbfed65559d0aca2: Status 404 returned error can't find the container with id c64e86def19e3b69629142ddf8fa2b37a772d20ac9862ad8dbfed65559d0aca2
Sep 16 12:20:57.919880 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:57.919838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-e80130" event={"ID":"82298439-954f-4a86-901e-b5798c5958a6","Type":"ContainerStarted","Data":"4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d"}
Sep 16 12:20:57.920298 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:57.919889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-e80130" event={"ID":"82298439-954f-4a86-901e-b5798c5958a6","Type":"ContainerStarted","Data":"c64e86def19e3b69629142ddf8fa2b37a772d20ac9862ad8dbfed65559d0aca2"}
Sep 16 12:20:57.936669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:57.936612    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7481-e80130" podStartSLOduration=1.9365757719999999 podStartE2EDuration="1.936575772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:57.934093274 +0000 UTC m=+1147.571418116" watchObservedRunningTime="2026-09-16 12:20:57.936575772 +0000 UTC m=+1147.573900611"
Sep 16 12:20:58.581054 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:58.580909    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-e80130"]
Sep 16 12:20:58.583754 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:58.583725    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-e80130"]
Sep 16 12:20:58.924914 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:20:58.924831    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7481-e80130" container="test-curl-7481-e80130"
Sep 16 12:20:59.931999 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:20:59.931931    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7481-e80130" podUID="82298439-954f-4a86-901e-b5798c5958a6" containerName="test-curl-7481-e80130" containerID="cri-o://4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d" gracePeriod=2
Sep 16 12:21:02.103268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.103242    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-e80130"
Sep 16 12:21:02.215456 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.215430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5fwr\" (UniqueName: \"kubernetes.io/projected/82298439-954f-4a86-901e-b5798c5958a6-kube-api-access-g5fwr\") pod \"82298439-954f-4a86-901e-b5798c5958a6\" (UID: \"82298439-954f-4a86-901e-b5798c5958a6\") "
Sep 16 12:21:02.218450 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.217459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82298439-954f-4a86-901e-b5798c5958a6-kube-api-access-g5fwr" (OuterVolumeSpecName: "kube-api-access-g5fwr") pod "82298439-954f-4a86-901e-b5798c5958a6" (UID: "82298439-954f-4a86-901e-b5798c5958a6"). InnerVolumeSpecName "kube-api-access-g5fwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:02.316210 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.316185    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5fwr\" (UniqueName: \"kubernetes.io/projected/82298439-954f-4a86-901e-b5798c5958a6-kube-api-access-g5fwr\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:21:02.884684 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.884648    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82298439-954f-4a86-901e-b5798c5958a6" path="/var/lib/kubelet/pods/82298439-954f-4a86-901e-b5798c5958a6/volumes"
Sep 16 12:21:02.951800 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.951771    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerID="5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf" exitCode=0
Sep 16 12:21:02.951971 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.951848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" event={"ID":"ba6fc64a-c1a9-41dd-aa41-4f08230fd589","Type":"ContainerDied","Data":"5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf"}
Sep 16 12:21:02.952308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.952289    2578 scope.go:117] "RemoveContainer" containerID="5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf"
Sep 16 12:21:02.953248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.953230    2578 generic.go:356] "Generic (PLEG): container finished" podID="82298439-954f-4a86-901e-b5798c5958a6" containerID="4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d" exitCode=137
Sep 16 12:21:02.953320 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.953288    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-e80130"
Sep 16 12:21:02.953320 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.953298    2578 scope.go:117] "RemoveContainer" containerID="4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d"
Sep 16 12:21:02.953648 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.953587    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:21:02.953773 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.953677    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:21:02.953773 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.953727    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:21:02.965525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.965489    2578 scope.go:117] "RemoveContainer" containerID="4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d"
Sep 16 12:21:02.965799 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:21:02.965782    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d\": container with ID starting with 4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d not found: ID does not exist" containerID="4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d"
Sep 16 12:21:02.965860 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:02.965807    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d"} err="failed to get container status \"4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d\": rpc error: code = NotFound desc = could not find container \"4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d\": container with ID starting with 4137b16290500fc6957f7b7aae4bcb06765e39e93521e02889a6fa6af84c828d not found: ID does not exist"
Sep 16 12:21:03.961000 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:03.960967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" event={"ID":"ba6fc64a-c1a9-41dd-aa41-4f08230fd589","Type":"ContainerStarted","Data":"22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c"}
Sep 16 12:21:04.340826 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:04.340722    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:21:04.340997 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:04.340915    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 12:21:04.340997 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:04.340978    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 12:21:05.339656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:05.339186    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 12:21:05.339656 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:05.339253    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 12:21:05.364548 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:05.364494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:21:05.364737 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:05.364574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:21:05.377129 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:05.377103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:21:05.786810 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:05.786774    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:21:05.986438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:05.986410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:21:06.339420 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:06.339380    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 12:21:06.339754 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:06.339719    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 12:21:07.340029 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:07.339980    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 12:21:07.340595 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:07.340055    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 12:21:08.339028 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:08.338974    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 12:21:08.339263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:08.339056    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 12:21:09.340207 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:09.340163    2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 16 12:21:09.340755 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:09.340233    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 16 12:21:10.351693 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.351658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:21:10.355142 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.355105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:21:10.463115 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.463071    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"]
Sep 16 12:21:10.463683 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.463663    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82298439-954f-4a86-901e-b5798c5958a6" containerName="test-curl-7481-e80130"
Sep 16 12:21:10.463740 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.463687    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="82298439-954f-4a86-901e-b5798c5958a6" containerName="test-curl-7481-e80130"
Sep 16 12:21:10.463823 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.463811    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="82298439-954f-4a86-901e-b5798c5958a6" containerName="test-curl-7481-e80130"
Sep 16 12:21:10.467192 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.467169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.469791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.469769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8f9e6d9e-serving-cert\""
Sep 16 12:21:10.478009 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.477986    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"]
Sep 16 12:21:10.611291 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.611203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eab0503-58fd-4217-89f6-f233da65f316-openshift-service-ca\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.611442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.611297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.611442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.611338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-metrics-certs\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.611442 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.611390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqr48\" (UniqueName: \"kubernetes.io/projected/7eab0503-58fd-4217-89f6-f233da65f316-kube-api-access-wqr48\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.712862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.712828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eab0503-58fd-4217-89f6-f233da65f316-openshift-service-ca\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.713037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.712888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.713037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.712930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-metrics-certs\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.713037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.712950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqr48\" (UniqueName: \"kubernetes.io/projected/7eab0503-58fd-4217-89f6-f233da65f316-kube-api-access-wqr48\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.713156 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:21:10.713063    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-serving-cert: secret "maas-api-e2e-mt-8f9e6d9e-serving-cert" not found
Sep 16 12:21:10.713156 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:21:10.713134    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls podName:7eab0503-58fd-4217-89f6-f233da65f316 nodeName:}" failed. No retries permitted until 2026-09-16 12:21:11.213115113 +0000 UTC m=+1160.850439942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls") pod "maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f" (UID: "7eab0503-58fd-4217-89f6-f233da65f316") : secret "maas-api-e2e-mt-8f9e6d9e-serving-cert" not found
Sep 16 12:21:10.713666 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.713642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eab0503-58fd-4217-89f6-f233da65f316-openshift-service-ca\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.715550 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.715506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-metrics-certs\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.730891 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.730869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqr48\" (UniqueName: \"kubernetes.io/projected/7eab0503-58fd-4217-89f6-f233da65f316-kube-api-access-wqr48\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:10.760993 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.760959    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"]
Sep 16 12:21:10.763506 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.763488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:10.766631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.766609    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"distinct-w0-ff057a-kserve-self-signed-certs\""
Sep 16 12:21:10.775025 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.775000    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"]
Sep 16 12:21:10.914551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.914440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-model-cache\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:10.914551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.914540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tmp-dir\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:10.914753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.914595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9fz\" (UniqueName: \"kubernetes.io/projected/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-kube-api-access-6t9fz\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:10.914753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.914616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tls-certs\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:10.914753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.914650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-home\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:10.914753 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:10.914686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-dshm\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.015923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.015888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tmp-dir\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.016092 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.015939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9fz\" (UniqueName: \"kubernetes.io/projected/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-kube-api-access-6t9fz\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.016135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.016104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tls-certs\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.016178 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.016142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-home\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.016232 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.016185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-dshm\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.016232 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.016220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-model-cache\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.016336 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.016287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tmp-dir\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.016486 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.016465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-home\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.016616 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.016600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-model-cache\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.018611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.018594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-dshm\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.018781 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.018766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tls-certs\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.025661 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.025637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9fz\" (UniqueName: \"kubernetes.io/projected/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-kube-api-access-6t9fz\") pod \"distinct-w0-ff057a-kserve-85f4ffd84-qw2ww\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.073704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.073672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:11.218771 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.218740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:11.221358 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.221334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls\") pod \"maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:11.355858 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.355552    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"]
Sep 16 12:21:11.359592 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:21:11.359565    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d631bc2_f9e2_4c68_b7a9_266ff1b3c85b.slice/crio-d0d2c24b91d3c34f5103487cedca73899a9d5680a8c0818450025a11edd89947 WatchSource:0}: Error finding container d0d2c24b91d3c34f5103487cedca73899a9d5680a8c0818450025a11edd89947: Status 404 returned error can't find the container with id d0d2c24b91d3c34f5103487cedca73899a9d5680a8c0818450025a11edd89947
Sep 16 12:21:11.381736 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.381713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:11.670167 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:11.670133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"]
Sep 16 12:21:11.671051 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:21:11.671026    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eab0503_58fd_4217_89f6_f233da65f316.slice/crio-f06e7ab5fe06825b610f860d939497a3430ff5d31b13a1c239a8ae2f8960036d WatchSource:0}: Error finding container f06e7ab5fe06825b610f860d939497a3430ff5d31b13a1c239a8ae2f8960036d: Status 404 returned error can't find the container with id f06e7ab5fe06825b610f860d939497a3430ff5d31b13a1c239a8ae2f8960036d
Sep 16 12:21:12.016574 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:12.016536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f" event={"ID":"7eab0503-58fd-4217-89f6-f233da65f316","Type":"ContainerStarted","Data":"f06e7ab5fe06825b610f860d939497a3430ff5d31b13a1c239a8ae2f8960036d"}
Sep 16 12:21:12.021726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:12.021700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww" event={"ID":"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b","Type":"ContainerStarted","Data":"4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df"}
Sep 16 12:21:12.021843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:12.021741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww" event={"ID":"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b","Type":"ContainerStarted","Data":"d0d2c24b91d3c34f5103487cedca73899a9d5680a8c0818450025a11edd89947"}
Sep 16 12:21:12.055539 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:12.050077    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww" podStartSLOduration=2.050052802 podStartE2EDuration="2.050052802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:12.046886293 +0000 UTC m=+1161.684211131" watchObservedRunningTime="2026-09-16 12:21:12.050052802 +0000 UTC m=+1161.687377641"
Sep 16 12:21:13.046471 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:13.046432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f" event={"ID":"7eab0503-58fd-4217-89f6-f233da65f316","Type":"ContainerStarted","Data":"9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248"}
Sep 16 12:21:13.046930 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:13.046821    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:13.067549 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:13.067482    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f" podStartSLOduration=2.728490838 podStartE2EDuration="3.067465818s" totalImagesPullingTime="338.97498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:13.065578029 +0000 UTC m=+1162.702902868" watchObservedRunningTime="2026-09-16 12:21:13.067465818 +0000 UTC m=+1162.704790698"
Sep 16 12:21:16.065847 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:16.065768    2578 generic.go:356] "Generic (PLEG): container finished" podID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerID="4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb" exitCode=0
Sep 16 12:21:16.066242 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:16.065837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" event={"ID":"63859a7d-a1da-45dd-abc0-95bdd0023b5c","Type":"ContainerDied","Data":"4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb"}
Sep 16 12:21:16.066242 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:16.066234    2578 scope.go:117] "RemoveContainer" containerID="4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb"
Sep 16 12:21:16.067494 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:16.067469    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:21:16.067585 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:16.067562    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:21:16.067627 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:16.067607    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:21:17.074608 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:17.074568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" event={"ID":"63859a7d-a1da-45dd-abc0-95bdd0023b5c","Type":"ContainerStarted","Data":"f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3"}
Sep 16 12:21:17.142234 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:17.142187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:21:17.143735 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:17.143705    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:17.143863 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:17.143765    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:21:17.340138 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:17.340042    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"]
Sep 16 12:21:18.142967 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:18.142918    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:18.143481 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:18.142993    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:21:18.870596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:18.870566    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:21:19.059373 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:19.059346    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:21:19.143966 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:19.143873    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:19.144551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:19.144498    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:21:20.142824 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:20.142371    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:20.142824 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:20.142432    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:21:21.074350 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:21.074311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:21.074864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:21.074558    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:21.087627 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:21.087605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:21.107660 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:21.107636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:21:21.144536 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:21.144487    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:21.144798 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:21.144769    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:21:22.143377 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:22.142944    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:22.143377 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:22.143005    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 16 12:21:23.146817 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:23.146789    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:21:23.147882 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:23.147861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:21:32.370598 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.370548    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p" podUID="eb62fdb3-3971-46a1-ba49-2d6851843606" containerName="main" containerID="cri-o://efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b" gracePeriod=45
Sep 16 12:21:32.635770 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.635704    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:21:32.773325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb62fdb3-3971-46a1-ba49-2d6851843606-tls-certs\") pod \"eb62fdb3-3971-46a1-ba49-2d6851843606\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") "
Sep 16 12:21:32.773488 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773332    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-model-cache\") pod \"eb62fdb3-3971-46a1-ba49-2d6851843606\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") "
Sep 16 12:21:32.773488 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773387    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7ztf\" (UniqueName: \"kubernetes.io/projected/eb62fdb3-3971-46a1-ba49-2d6851843606-kube-api-access-j7ztf\") pod \"eb62fdb3-3971-46a1-ba49-2d6851843606\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") "
Sep 16 12:21:32.773488 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773413    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-tmp-dir\") pod \"eb62fdb3-3971-46a1-ba49-2d6851843606\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") "
Sep 16 12:21:32.773488 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773438    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-home\") pod \"eb62fdb3-3971-46a1-ba49-2d6851843606\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") "
Sep 16 12:21:32.773488 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773482    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-dshm\") pod \"eb62fdb3-3971-46a1-ba49-2d6851843606\" (UID: \"eb62fdb3-3971-46a1-ba49-2d6851843606\") "
Sep 16 12:21:32.773791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb62fdb3-3971-46a1-ba49-2d6851843606" (UID: "eb62fdb3-3971-46a1-ba49-2d6851843606"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:32.773791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb62fdb3-3971-46a1-ba49-2d6851843606" (UID: "eb62fdb3-3971-46a1-ba49-2d6851843606"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:32.773791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773744    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-home" (OuterVolumeSpecName: "home") pod "eb62fdb3-3971-46a1-ba49-2d6851843606" (UID: "eb62fdb3-3971-46a1-ba49-2d6851843606"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:32.773919 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773878    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:21:32.773919 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773913    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:21:32.773987 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.773923    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:21:32.775777 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.775750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-dshm" (OuterVolumeSpecName: "dshm") pod "eb62fdb3-3971-46a1-ba49-2d6851843606" (UID: "eb62fdb3-3971-46a1-ba49-2d6851843606"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:32.775876 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.775782    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb62fdb3-3971-46a1-ba49-2d6851843606-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb62fdb3-3971-46a1-ba49-2d6851843606" (UID: "eb62fdb3-3971-46a1-ba49-2d6851843606"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:21:32.775876 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.775855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb62fdb3-3971-46a1-ba49-2d6851843606-kube-api-access-j7ztf" (OuterVolumeSpecName: "kube-api-access-j7ztf") pod "eb62fdb3-3971-46a1-ba49-2d6851843606" (UID: "eb62fdb3-3971-46a1-ba49-2d6851843606"). InnerVolumeSpecName "kube-api-access-j7ztf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:32.875213 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.875182    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7ztf\" (UniqueName: \"kubernetes.io/projected/eb62fdb3-3971-46a1-ba49-2d6851843606-kube-api-access-j7ztf\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:21:32.875213 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.875207    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb62fdb3-3971-46a1-ba49-2d6851843606-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:21:32.875213 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:32.875217    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb62fdb3-3971-46a1-ba49-2d6851843606-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:21:33.177551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.177439    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb62fdb3-3971-46a1-ba49-2d6851843606" containerID="efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b" exitCode=0
Sep 16 12:21:33.177551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.177497    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"
Sep 16 12:21:33.177788 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.177550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p" event={"ID":"eb62fdb3-3971-46a1-ba49-2d6851843606","Type":"ContainerDied","Data":"efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b"}
Sep 16 12:21:33.177788 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.177596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p" event={"ID":"eb62fdb3-3971-46a1-ba49-2d6851843606","Type":"ContainerDied","Data":"5ffe745e395489ef72be70e026e3b40f5d7f12402032c06cd0bc10a1005a319f"}
Sep 16 12:21:33.177788 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.177613    2578 scope.go:117] "RemoveContainer" containerID="efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b"
Sep 16 12:21:33.186915 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.186895    2578 scope.go:117] "RemoveContainer" containerID="efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b"
Sep 16 12:21:33.187184 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:21:33.187167    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b\": container with ID starting with efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b not found: ID does not exist" containerID="efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b"
Sep 16 12:21:33.187238 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.187196    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b"} err="failed to get container status \"efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b\": rpc error: code = NotFound desc = could not find container \"efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b\": container with ID starting with efebaa22ce7033d7274057bde8b17e6710a215b62302b5e489b166eec10da46b not found: ID does not exist"
Sep 16 12:21:33.199924 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.199898    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"]
Sep 16 12:21:33.205475 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:33.205452    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c26b8c/noten-model-c26b8c-kserve-bb6cf5467-rd22p"]
Sep 16 12:21:34.884557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:34.884504    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb62fdb3-3971-46a1-ba49-2d6851843606" path="/var/lib/kubelet/pods/eb62fdb3-3971-46a1-ba49-2d6851843606/volumes"
Sep 16 12:21:50.897889 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:50.897863    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:21:50.900427 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:21:50.900404    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:22:33.569938 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.569830    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"]
Sep 16 12:22:33.571037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.571012    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb62fdb3-3971-46a1-ba49-2d6851843606" containerName="main"
Sep 16 12:22:33.571232 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.571195    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb62fdb3-3971-46a1-ba49-2d6851843606" containerName="main"
Sep 16 12:22:33.571430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.571412    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb62fdb3-3971-46a1-ba49-2d6851843606" containerName="main"
Sep 16 12:22:33.574870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.574848    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"]
Sep 16 12:22:33.574994 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.574934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.578025 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.578002    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"distinct-w4-556077-kserve-self-signed-certs\""
Sep 16 12:22:33.653466 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.653436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-model-cache\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.653649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.653485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-home\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.653649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.653504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-tmp-dir\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.653649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.653584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-dshm\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.653649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.653644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a4350-37cd-43d1-bfc7-0a8084345aed-tls-certs\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.653816 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.653683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscdt\" (UniqueName: \"kubernetes.io/projected/778a4350-37cd-43d1-bfc7-0a8084345aed-kube-api-access-nscdt\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.754360 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.754329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a4350-37cd-43d1-bfc7-0a8084345aed-tls-certs\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.754588 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.754376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscdt\" (UniqueName: \"kubernetes.io/projected/778a4350-37cd-43d1-bfc7-0a8084345aed-kube-api-access-nscdt\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.754588 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.754418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-model-cache\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.754588 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.754456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-home\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.754588 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.754472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-tmp-dir\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.754588 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.754496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-dshm\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.754936 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.754916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-model-cache\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.754989 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.754974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-tmp-dir\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.755026 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.755006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-home\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.756969 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.756944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-dshm\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.757258 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.757238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a4350-37cd-43d1-bfc7-0a8084345aed-tls-certs\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.762756 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.762733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscdt\" (UniqueName: \"kubernetes.io/projected/778a4350-37cd-43d1-bfc7-0a8084345aed-kube-api-access-nscdt\") pod \"distinct-w4-556077-kserve-678b85cc45-pjtw5\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") " pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:33.885200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:33.885117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:34.157600 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:34.157565    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"]
Sep 16 12:22:34.158654 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:22:34.158628    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod778a4350_37cd_43d1_bfc7_0a8084345aed.slice/crio-1a8b7e56cae7c036b0a64b0ef0fbabd558234e8fac27043eef806635ac59bc54 WatchSource:0}: Error finding container 1a8b7e56cae7c036b0a64b0ef0fbabd558234e8fac27043eef806635ac59bc54: Status 404 returned error can't find the container with id 1a8b7e56cae7c036b0a64b0ef0fbabd558234e8fac27043eef806635ac59bc54
Sep 16 12:22:34.521697 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:34.521662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5" event={"ID":"778a4350-37cd-43d1-bfc7-0a8084345aed","Type":"ContainerStarted","Data":"421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b"}
Sep 16 12:22:34.521697 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:34.521701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5" event={"ID":"778a4350-37cd-43d1-bfc7-0a8084345aed","Type":"ContainerStarted","Data":"1a8b7e56cae7c036b0a64b0ef0fbabd558234e8fac27043eef806635ac59bc54"}
Sep 16 12:22:34.541790 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:34.541740    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5" podStartSLOduration=1.541724684 podStartE2EDuration="1.541724684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:34.539884604 +0000 UTC m=+1244.177209444" watchObservedRunningTime="2026-09-16 12:22:34.541724684 +0000 UTC m=+1244.179049576"
Sep 16 12:22:43.885324 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:43.885230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:43.885324 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:43.885274    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:43.897657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:43.897630    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:44.588310 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:44.588282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:22:59.653496 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.653457    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"]
Sep 16 12:22:59.657125 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.657103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.661825 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.661799    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8f9e6d9e\"/\"e2e-lifecycle-model-8f9e6d9e-kserve-self-signed-certs\""
Sep 16 12:22:59.662151 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.661844    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8f9e6d9e\"/\"openshift-service-ca.crt\""
Sep 16 12:22:59.662456 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.661899    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8f9e6d9e\"/\"kube-root-ca.crt\""
Sep 16 12:22:59.662634 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.662611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8f9e6d9e\"/\"default-dockercfg-lhwp8\""
Sep 16 12:22:59.666351 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.666321    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"]
Sep 16 12:22:59.692182 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.692155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gphb6\" (UniqueName: \"kubernetes.io/projected/c4938922-fb61-4829-8f5e-e0b34afd0bb5-kube-api-access-gphb6\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.692309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.692213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-home\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.692309 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.692288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tls-certs\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.692385 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.692331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tmp-dir\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.692385 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.692368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-dshm\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.692452 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.692423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-model-cache\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.793671 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.793644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-dshm\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.793812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.793689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-model-cache\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.793812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.793726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gphb6\" (UniqueName: \"kubernetes.io/projected/c4938922-fb61-4829-8f5e-e0b34afd0bb5-kube-api-access-gphb6\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.793812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.793765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-home\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.793812 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.793793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tls-certs\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.794034 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.793822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tmp-dir\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.794172 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.794151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-model-cache\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.794228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.794204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-home\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.794266 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.794222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tmp-dir\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.796104 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.796087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-dshm\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.796352 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.796333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tls-certs\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.802727 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.802701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gphb6\" (UniqueName: \"kubernetes.io/projected/c4938922-fb61-4829-8f5e-e0b34afd0bb5-kube-api-access-gphb6\") pod \"e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") " pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:22:59.970539 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:22:59.970488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:23:00.245798 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:23:00.245769    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"]
Sep 16 12:23:00.246666 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:23:00.246639    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4938922_fb61_4829_8f5e_e0b34afd0bb5.slice/crio-c9b12725caa780a80e1cde01c7624d074ccebcf98784d9e9bec7bb7b70fad7aa WatchSource:0}: Error finding container c9b12725caa780a80e1cde01c7624d074ccebcf98784d9e9bec7bb7b70fad7aa: Status 404 returned error can't find the container with id c9b12725caa780a80e1cde01c7624d074ccebcf98784d9e9bec7bb7b70fad7aa
Sep 16 12:23:00.669484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:23:00.669389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd" event={"ID":"c4938922-fb61-4829-8f5e-e0b34afd0bb5","Type":"ContainerStarted","Data":"de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971"}
Sep 16 12:23:00.669484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:23:00.669440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd" event={"ID":"c4938922-fb61-4829-8f5e-e0b34afd0bb5","Type":"ContainerStarted","Data":"c9b12725caa780a80e1cde01c7624d074ccebcf98784d9e9bec7bb7b70fad7aa"}
Sep 16 12:23:00.689721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:23:00.689669    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd" podStartSLOduration=1.6896546159999999 podStartE2EDuration="1.689654616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:00.68772563 +0000 UTC m=+1270.325050469" watchObservedRunningTime="2026-09-16 12:23:00.689654616 +0000 UTC m=+1270.326979453"
Sep 16 12:23:09.970860 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:23:09.970818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:23:09.971463 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:23:09.970963    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:23:09.983639 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:23:09.983615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:23:10.736626 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:23:10.736597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:24:28.546764 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.546685    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"]
Sep 16 12:24:28.550847 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.550824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.553307 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.553284    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"distinct-2-w4-556077-kserve-self-signed-certs\""
Sep 16 12:24:28.558337 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.558315    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"]
Sep 16 12:24:28.680010 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.679972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-model-cache\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.680010 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.680024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tmp-dir\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.680290 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.680050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-dshm\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.680290 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.680067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tls-certs\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.680290 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.680152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td2vc\" (UniqueName: \"kubernetes.io/projected/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-kube-api-access-td2vc\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.680290 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.680221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-home\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781353 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-home\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-model-cache\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tmp-dir\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-dshm\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tls-certs\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781784 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td2vc\" (UniqueName: \"kubernetes.io/projected/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-kube-api-access-td2vc\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781784 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-home\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tmp-dir\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.781886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.781856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-model-cache\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.783976 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.783959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-dshm\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.784080 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.784063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tls-certs\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.789864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.789845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td2vc\" (UniqueName: \"kubernetes.io/projected/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-kube-api-access-td2vc\") pod \"distinct-2-w4-556077-kserve-59b6674bd6-22mq6\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") " pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:28.863146 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:28.863082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:29.136592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:29.136568    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"]
Sep 16 12:24:29.138155 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:24:29.138125    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84800a49_1cb0_47c4_ace3_d9e6b5e4cda4.slice/crio-974bafbd04e0faa90ef43c0455e811a100c1bd3f12d14ad163de0e22d3453235 WatchSource:0}: Error finding container 974bafbd04e0faa90ef43c0455e811a100c1bd3f12d14ad163de0e22d3453235: Status 404 returned error can't find the container with id 974bafbd04e0faa90ef43c0455e811a100c1bd3f12d14ad163de0e22d3453235
Sep 16 12:24:29.164748 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:29.164722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6" event={"ID":"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4","Type":"ContainerStarted","Data":"974bafbd04e0faa90ef43c0455e811a100c1bd3f12d14ad163de0e22d3453235"}
Sep 16 12:24:30.172916 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:30.172883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6" event={"ID":"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4","Type":"ContainerStarted","Data":"17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3"}
Sep 16 12:24:30.190232 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:30.190188    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6" podStartSLOduration=2.190172653 podStartE2EDuration="2.190172653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:30.188193932 +0000 UTC m=+1359.825518768" watchObservedRunningTime="2026-09-16 12:24:30.190172653 +0000 UTC m=+1359.827497490"
Sep 16 12:24:36.476736 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.476702    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"]
Sep 16 12:24:36.477281 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.476993    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f" podUID="7eab0503-58fd-4217-89f6-f233da65f316" containerName="maas-api" containerID="cri-o://9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248" gracePeriod=30
Sep 16 12:24:36.730563 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.730479    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:24:36.867388 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.867352    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eab0503-58fd-4217-89f6-f233da65f316-openshift-service-ca\") pod \"7eab0503-58fd-4217-89f6-f233da65f316\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") "
Sep 16 12:24:36.867582 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.867483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls\") pod \"7eab0503-58fd-4217-89f6-f233da65f316\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") "
Sep 16 12:24:36.867582 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.867541    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqr48\" (UniqueName: \"kubernetes.io/projected/7eab0503-58fd-4217-89f6-f233da65f316-kube-api-access-wqr48\") pod \"7eab0503-58fd-4217-89f6-f233da65f316\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") "
Sep 16 12:24:36.867684 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.867586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-metrics-certs\") pod \"7eab0503-58fd-4217-89f6-f233da65f316\" (UID: \"7eab0503-58fd-4217-89f6-f233da65f316\") "
Sep 16 12:24:36.868038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.867794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eab0503-58fd-4217-89f6-f233da65f316-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7eab0503-58fd-4217-89f6-f233da65f316" (UID: "7eab0503-58fd-4217-89f6-f233da65f316"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:24:36.868186 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.868123    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eab0503-58fd-4217-89f6-f233da65f316-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:24:36.869874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.869850    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7eab0503-58fd-4217-89f6-f233da65f316" (UID: "7eab0503-58fd-4217-89f6-f233da65f316"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:24:36.870372 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.870349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7eab0503-58fd-4217-89f6-f233da65f316" (UID: "7eab0503-58fd-4217-89f6-f233da65f316"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:24:36.870439 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.870376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eab0503-58fd-4217-89f6-f233da65f316-kube-api-access-wqr48" (OuterVolumeSpecName: "kube-api-access-wqr48") pod "7eab0503-58fd-4217-89f6-f233da65f316" (UID: "7eab0503-58fd-4217-89f6-f233da65f316"). InnerVolumeSpecName "kube-api-access-wqr48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:24:36.969705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.969674    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:24:36.969705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.969702    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqr48\" (UniqueName: \"kubernetes.io/projected/7eab0503-58fd-4217-89f6-f233da65f316-kube-api-access-wqr48\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:24:36.969883 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:36.969715    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eab0503-58fd-4217-89f6-f233da65f316-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:24:37.212797 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.212766    2578 generic.go:356] "Generic (PLEG): container finished" podID="7eab0503-58fd-4217-89f6-f233da65f316" containerID="9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248" exitCode=0
Sep 16 12:24:37.212985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.212814    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"
Sep 16 12:24:37.212985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.212852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f" event={"ID":"7eab0503-58fd-4217-89f6-f233da65f316","Type":"ContainerDied","Data":"9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248"}
Sep 16 12:24:37.212985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.212905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f" event={"ID":"7eab0503-58fd-4217-89f6-f233da65f316","Type":"ContainerDied","Data":"f06e7ab5fe06825b610f860d939497a3430ff5d31b13a1c239a8ae2f8960036d"}
Sep 16 12:24:37.212985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.212928    2578 scope.go:117] "RemoveContainer" containerID="9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248"
Sep 16 12:24:37.222450 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.222433    2578 scope.go:117] "RemoveContainer" containerID="9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248"
Sep 16 12:24:37.222771 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:24:37.222749    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248\": container with ID starting with 9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248 not found: ID does not exist" containerID="9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248"
Sep 16 12:24:37.222852 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.222783    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248"} err="failed to get container status \"9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248\": rpc error: code = NotFound desc = could not find container \"9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248\": container with ID starting with 9643baee6efb0645fc2526196e70475dbaf3c0cd67d61d75c45f24def4c71248 not found: ID does not exist"
Sep 16 12:24:37.230652 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.230616    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"]
Sep 16 12:24:37.233560 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:37.233538    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f9e6d9e-696d7bf797-8kb9f"]
Sep 16 12:24:38.863254 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:38.863221    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:38.863704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:38.863275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:38.875750 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:38.875721    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:38.884420 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:38.884398    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eab0503-58fd-4217-89f6-f233da65f316" path="/var/lib/kubelet/pods/7eab0503-58fd-4217-89f6-f233da65f316/volumes"
Sep 16 12:24:39.235931 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:39.235898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:24:44.885569 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:44.885534    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"]
Sep 16 12:24:59.905012 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:24:59.904947    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd" podUID="c4938922-fb61-4829-8f5e-e0b34afd0bb5" containerName="main" containerID="cri-o://de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971" gracePeriod=45
Sep 16 12:25:00.164728 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.164670    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:25:00.189271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.189240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tmp-dir\") pod \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") "
Sep 16 12:25:00.189430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.189297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-model-cache\") pod \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") "
Sep 16 12:25:00.189430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.189335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-dshm\") pod \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") "
Sep 16 12:25:00.189430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.189394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-home\") pod \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") "
Sep 16 12:25:00.189622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.189459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gphb6\" (UniqueName: \"kubernetes.io/projected/c4938922-fb61-4829-8f5e-e0b34afd0bb5-kube-api-access-gphb6\") pod \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") "
Sep 16 12:25:00.189622 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.189576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tls-certs\") pod \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\" (UID: \"c4938922-fb61-4829-8f5e-e0b34afd0bb5\") "
Sep 16 12:25:00.192381 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.192353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4938922-fb61-4829-8f5e-e0b34afd0bb5" (UID: "c4938922-fb61-4829-8f5e-e0b34afd0bb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:00.192568 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.192405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-home" (OuterVolumeSpecName: "home") pod "c4938922-fb61-4829-8f5e-e0b34afd0bb5" (UID: "c4938922-fb61-4829-8f5e-e0b34afd0bb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:00.193106 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.192801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4938922-fb61-4829-8f5e-e0b34afd0bb5" (UID: "c4938922-fb61-4829-8f5e-e0b34afd0bb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:00.194104 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.193579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4938922-fb61-4829-8f5e-e0b34afd0bb5" (UID: "c4938922-fb61-4829-8f5e-e0b34afd0bb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:00.197052 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.197009    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-dshm" (OuterVolumeSpecName: "dshm") pod "c4938922-fb61-4829-8f5e-e0b34afd0bb5" (UID: "c4938922-fb61-4829-8f5e-e0b34afd0bb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:00.198454 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.197462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4938922-fb61-4829-8f5e-e0b34afd0bb5-kube-api-access-gphb6" (OuterVolumeSpecName: "kube-api-access-gphb6") pod "c4938922-fb61-4829-8f5e-e0b34afd0bb5" (UID: "c4938922-fb61-4829-8f5e-e0b34afd0bb5"). InnerVolumeSpecName "kube-api-access-gphb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:00.291678 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.291647    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gphb6\" (UniqueName: \"kubernetes.io/projected/c4938922-fb61-4829-8f5e-e0b34afd0bb5-kube-api-access-gphb6\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:25:00.291678 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.291676    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:25:00.291678 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.291688    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:25:00.291923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.291700    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:25:00.291923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.291711    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:25:00.291923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.291717    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4938922-fb61-4829-8f5e-e0b34afd0bb5-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:25:00.349283 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.349241    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4938922-fb61-4829-8f5e-e0b34afd0bb5" containerID="de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971" exitCode=0
Sep 16 12:25:00.349421 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.349321    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"
Sep 16 12:25:00.349421 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.349319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd" event={"ID":"c4938922-fb61-4829-8f5e-e0b34afd0bb5","Type":"ContainerDied","Data":"de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971"}
Sep 16 12:25:00.349525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.349433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd" event={"ID":"c4938922-fb61-4829-8f5e-e0b34afd0bb5","Type":"ContainerDied","Data":"c9b12725caa780a80e1cde01c7624d074ccebcf98784d9e9bec7bb7b70fad7aa"}
Sep 16 12:25:00.349525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.349450    2578 scope.go:117] "RemoveContainer" containerID="de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971"
Sep 16 12:25:00.359386 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.359368    2578 scope.go:117] "RemoveContainer" containerID="de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971"
Sep 16 12:25:00.359697 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:25:00.359674    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971\": container with ID starting with de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971 not found: ID does not exist" containerID="de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971"
Sep 16 12:25:00.359764 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.359703    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971"} err="failed to get container status \"de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971\": rpc error: code = NotFound desc = could not find container \"de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971\": container with ID starting with de2a76ea03353721d40c3642ec28b20c76621c288e6dc6bf2afe554a1d6e7971 not found: ID does not exist"
Sep 16 12:25:00.371649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.371625    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"]
Sep 16 12:25:00.375059 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.375036    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8f9e6d9e/e2e-lifecycle-model-8f9e6d9e-kserve-f889849c9-r8hpd"]
Sep 16 12:25:00.886280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:25:00.886239    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4938922-fb61-4829-8f5e-e0b34afd0bb5" path="/var/lib/kubelet/pods/c4938922-fb61-4829-8f5e-e0b34afd0bb5/volumes"
Sep 16 12:26:23.808923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.808892    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"]
Sep 16 12:26:23.811413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.809322    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eab0503-58fd-4217-89f6-f233da65f316" containerName="maas-api"
Sep 16 12:26:23.811413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.809336    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eab0503-58fd-4217-89f6-f233da65f316" containerName="maas-api"
Sep 16 12:26:23.811413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.809348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4938922-fb61-4829-8f5e-e0b34afd0bb5" containerName="main"
Sep 16 12:26:23.811413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.809354    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4938922-fb61-4829-8f5e-e0b34afd0bb5" containerName="main"
Sep 16 12:26:23.811413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.809427    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eab0503-58fd-4217-89f6-f233da65f316" containerName="maas-api"
Sep 16 12:26:23.811413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.809435    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4938922-fb61-4829-8f5e-e0b34afd0bb5" containerName="main"
Sep 16 12:26:23.812362 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.812341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.815032 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.815014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a6f57e-openshift-default-dockercfg-4bkgb\""
Sep 16 12:26:23.825750 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.825724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"]
Sep 16 12:26:23.919843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.919812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2fe014-795f-490e-8920-a7f8a9e9889d-istiod-ca-cert\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.920023 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.919857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxt2\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-kube-api-access-phxt2\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.920023 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.919903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-data\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.920023 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.919933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-credential-socket\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.920023 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.919994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-podinfo\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.920237 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.920073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-envoy\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.920237 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.920121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-socket\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.920237 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.920145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-certs\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:23.920237 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:23.920164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-token\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.020980 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.020948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-token\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021165 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.020997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2fe014-795f-490e-8920-a7f8a9e9889d-istiod-ca-cert\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021165 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxt2\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-kube-api-access-phxt2\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021165 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-data\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021165 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-credential-socket\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021412 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-podinfo\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021412 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-envoy\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021412 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-socket\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021412 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-certs\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-data\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021738 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-certs\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.021849 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2fe014-795f-490e-8920-a7f8a9e9889d-istiod-ca-cert\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.022046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.021991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-credential-socket\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.022267 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.022204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-socket\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.024631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.024607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-envoy\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.024956 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.024935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-podinfo\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.030989 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.030958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-token\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.031896 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.031872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxt2\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-kube-api-access-phxt2\") pod \"e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") " pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.123842 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.123758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:24.405864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.405835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"]
Sep 16 12:26:24.406624 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:26:24.406592    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2fe014_795f_490e_8920_a7f8a9e9889d.slice/crio-03381e1e01b9db3f5c7213d0fe52156dc5256beb776093e49fe85df58f7331e6 WatchSource:0}: Error finding container 03381e1e01b9db3f5c7213d0fe52156dc5256beb776093e49fe85df58f7331e6: Status 404 returned error can't find the container with id 03381e1e01b9db3f5c7213d0fe52156dc5256beb776093e49fe85df58f7331e6
Sep 16 12:26:24.408858 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.408828    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:24.408976 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.408886    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:24.408976 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.408915    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:24.820921 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.820892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" event={"ID":"3e2fe014-795f-490e-8920-a7f8a9e9889d","Type":"ContainerStarted","Data":"95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64"}
Sep 16 12:26:24.821305 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.820944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" event={"ID":"3e2fe014-795f-490e-8920-a7f8a9e9889d","Type":"ContainerStarted","Data":"03381e1e01b9db3f5c7213d0fe52156dc5256beb776093e49fe85df58f7331e6"}
Sep 16 12:26:24.842415 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:24.842369    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podStartSLOduration=1.842354583 podStartE2EDuration="1.842354583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:24.839842296 +0000 UTC m=+1474.477167135" watchObservedRunningTime="2026-09-16 12:26:24.842354583 +0000 UTC m=+1474.479679477"
Sep 16 12:26:25.124637 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:25.124562    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:25.129577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:25.129554    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:25.827259 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:25.827226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:25.828089 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:25.828069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:50.961108 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:50.961075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:26:50.965907 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:50.965886    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:26:55.829302 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:55.829261    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:26:55.829757 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:55.829332    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 16 12:26:56.010752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:56.010720    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerID="95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64" exitCode=0
Sep 16 12:26:56.010947 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:56.010800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" event={"ID":"3e2fe014-795f-490e-8920-a7f8a9e9889d","Type":"ContainerDied","Data":"95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64"}
Sep 16 12:26:56.011231 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:56.011217    2578 scope.go:117] "RemoveContainer" containerID="95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64"
Sep 16 12:26:56.012588 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:56.012551    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:56.012713 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:56.012646    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:56.012713 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:56.012695    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:57.017979 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.017946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" event={"ID":"3e2fe014-795f-490e-8920-a7f8a9e9889d","Type":"ContainerStarted","Data":"9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6"}
Sep 16 12:26:57.124099 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.124071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:26:57.124259 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.124228    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:26:57.124303 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.124284    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 16 12:26:57.794114 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.794080    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"]
Sep 16 12:26:57.796929 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.796911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.799985 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.799965    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-30f927-openshift-default-dockercfg-6cgpf\""
Sep 16 12:26:57.812857 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.812836    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"]
Sep 16 12:26:57.864017 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.863977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-socket\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.864017 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.864018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-data\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.864271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.864043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-token\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.864271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.864061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-envoy\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.864271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.864095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-certs\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.864271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.864130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-podinfo\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.864271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.864219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264cb7cc-88e8-4f3b-bb28-28eea673481d-istiod-ca-cert\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.864271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.864239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw97x\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-kube-api-access-hw97x\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.864271 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.864261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-credential-socket\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.965578 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.965501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264cb7cc-88e8-4f3b-bb28-28eea673481d-istiod-ca-cert\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.965802 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.965591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw97x\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-kube-api-access-hw97x\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.965870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.965796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-credential-socket\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.965910 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.965870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-socket\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.965910 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.965903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-data\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.965982 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.965943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-token\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.966031 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.965988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-envoy\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.966092 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.966049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-certs\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.966153 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.966105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-podinfo\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.966214 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.966157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-credential-socket\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.966276 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.966258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-socket\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.966466 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.966442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264cb7cc-88e8-4f3b-bb28-28eea673481d-istiod-ca-cert\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.966466 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.966458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-data\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.966610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.966496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-certs\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.968476 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.968451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-envoy\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:57.968638 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:57.968622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-podinfo\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:58.002589 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.002562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw97x\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-kube-api-access-hw97x\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:58.002767 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.002750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-token\") pod \"e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") " pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:58.107057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.106969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:58.125097 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.125063    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:26:58.125204 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.125142    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 16 12:26:58.407470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.407445    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"]
Sep 16 12:26:58.407991 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:26:58.407967    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264cb7cc_88e8_4f3b_bb28_28eea673481d.slice/crio-87a0304b7040123bd1e935f4f68bc49b07679383e23bf3add70400b097d6ab86 WatchSource:0}: Error finding container 87a0304b7040123bd1e935f4f68bc49b07679383e23bf3add70400b097d6ab86: Status 404 returned error can't find the container with id 87a0304b7040123bd1e935f4f68bc49b07679383e23bf3add70400b097d6ab86
Sep 16 12:26:58.410485 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.410455    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:58.410582 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.410545    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:58.410582 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:58.410576    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:26:59.031653 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:59.031611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" event={"ID":"264cb7cc-88e8-4f3b-bb28-28eea673481d","Type":"ContainerStarted","Data":"698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886"}
Sep 16 12:26:59.031858 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:59.031683    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" event={"ID":"264cb7cc-88e8-4f3b-bb28-28eea673481d","Type":"ContainerStarted","Data":"87a0304b7040123bd1e935f4f68bc49b07679383e23bf3add70400b097d6ab86"}
Sep 16 12:26:59.059268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:59.059217    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" podStartSLOduration=2.059200835 podStartE2EDuration="2.059200835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:59.055532197 +0000 UTC m=+1508.692857030" watchObservedRunningTime="2026-09-16 12:26:59.059200835 +0000 UTC m=+1508.696525672"
Sep 16 12:26:59.107847 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:59.107814    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:59.113107 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:59.113083    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:26:59.125081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:59.125050    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:26:59.125191 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:26:59.125108    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 16 12:27:00.043311 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:00.043275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:27:00.045187 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:00.045148    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:27:00.124896 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:00.124859    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:27:00.125265 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:00.124932    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 16 12:27:01.124959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:01.124922    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:27:01.125425 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:01.124983    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 16 12:27:02.124150 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:02.124118    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:27:02.124335 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:02.124174    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 16 12:27:03.124658 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:03.124619    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:27:03.125138 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:03.124685    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 16 12:27:04.129352 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:04.129324    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:27:04.129830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:04.129631    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:27:04.130245 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:04.130229    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:27:06.571189 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.571158    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"]
Sep 16 12:27:06.573972 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.573951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.576980 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.576960    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-30f927-serving-cert\""
Sep 16 12:27:06.587739 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.587700    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"]
Sep 16 12:27:06.658026 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.657984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r92x\" (UniqueName: \"kubernetes.io/projected/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-kube-api-access-7r92x\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.658214 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.658051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-maas-api-tls\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.658214 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.658085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-metrics-certs\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.658214 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.658199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-openshift-service-ca\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.759555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.759506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-openshift-service-ca\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.759789 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.759626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r92x\" (UniqueName: \"kubernetes.io/projected/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-kube-api-access-7r92x\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.759789 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.759649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-maas-api-tls\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.759789 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.759668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-metrics-certs\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.760296 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.760270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-openshift-service-ca\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.762234 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.762211    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-metrics-certs\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.762355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.762256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-maas-api-tls\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.768073 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.768046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r92x\" (UniqueName: \"kubernetes.io/projected/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-kube-api-access-7r92x\") pod \"maas-api-e2e-api-b-30f927-649b66d8b-sp2r6\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:06.887320 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:06.887238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:07.161440 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:07.161405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"]
Sep 16 12:27:07.162853 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:27:07.162813    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdda458_958c_4ea0_a2d9_6b4e71b665cf.slice/crio-68493b44d2a3117968a5856c8a2d10acad9ba6a881d85023720591497f791689 WatchSource:0}: Error finding container 68493b44d2a3117968a5856c8a2d10acad9ba6a881d85023720591497f791689: Status 404 returned error can't find the container with id 68493b44d2a3117968a5856c8a2d10acad9ba6a881d85023720591497f791689
Sep 16 12:27:07.173201 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:07.173127    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:27:08.096037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:08.095993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6" event={"ID":"6bdda458-958c-4ea0-a2d9-6b4e71b665cf","Type":"ContainerStarted","Data":"fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17"}
Sep 16 12:27:08.096557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:08.096098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6" event={"ID":"6bdda458-958c-4ea0-a2d9-6b4e71b665cf","Type":"ContainerStarted","Data":"68493b44d2a3117968a5856c8a2d10acad9ba6a881d85023720591497f791689"}
Sep 16 12:27:08.096557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:08.096130    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:08.119602 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:08.119487    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6" podStartSLOduration=1.439902638 podStartE2EDuration="2.119473527s" totalImagesPullingTime="679.570889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:08.11575692 +0000 UTC m=+1517.753081758" watchObservedRunningTime="2026-09-16 12:27:08.119473527 +0000 UTC m=+1517.756798411"
Sep 16 12:27:13.283520 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:13.283427    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"]
Sep 16 12:27:13.293896 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:13.293857    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"]
Sep 16 12:27:13.301276 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:13.301245    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"]
Sep 16 12:27:13.315787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:13.315762    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"]
Sep 16 12:27:14.108360 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:14.108334    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:27:26.971931 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:26.971898    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"]
Sep 16 12:27:26.972583 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:26.972110    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b" podUID="68d0dc60-dd05-494b-b341-b97877d98ba4" containerName="maas-api" containerID="cri-o://6188aca2029ea8ac94b7c21214fe0cdbea6c8be031d9683369084d4cc6b3b138" gracePeriod=30
Sep 16 12:27:27.222355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.222267    2578 generic.go:356] "Generic (PLEG): container finished" podID="68d0dc60-dd05-494b-b341-b97877d98ba4" containerID="6188aca2029ea8ac94b7c21214fe0cdbea6c8be031d9683369084d4cc6b3b138" exitCode=0
Sep 16 12:27:27.222355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.222328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b" event={"ID":"68d0dc60-dd05-494b-b341-b97877d98ba4","Type":"ContainerDied","Data":"6188aca2029ea8ac94b7c21214fe0cdbea6c8be031d9683369084d4cc6b3b138"}
Sep 16 12:27:27.283826 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.283795    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:27:27.371361 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.371326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0dc60-dd05-494b-b341-b97877d98ba4-openshift-service-ca\") pod \"68d0dc60-dd05-494b-b341-b97877d98ba4\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") "
Sep 16 12:27:27.371577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.371421    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-maas-api-tls\") pod \"68d0dc60-dd05-494b-b341-b97877d98ba4\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") "
Sep 16 12:27:27.371577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.371498    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vldw\" (UniqueName: \"kubernetes.io/projected/68d0dc60-dd05-494b-b341-b97877d98ba4-kube-api-access-7vldw\") pod \"68d0dc60-dd05-494b-b341-b97877d98ba4\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") "
Sep 16 12:27:27.371577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.371563    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-metrics-certs\") pod \"68d0dc60-dd05-494b-b341-b97877d98ba4\" (UID: \"68d0dc60-dd05-494b-b341-b97877d98ba4\") "
Sep 16 12:27:27.371804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.371771    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d0dc60-dd05-494b-b341-b97877d98ba4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68d0dc60-dd05-494b-b341-b97877d98ba4" (UID: "68d0dc60-dd05-494b-b341-b97877d98ba4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:27.373923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.373895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68d0dc60-dd05-494b-b341-b97877d98ba4" (UID: "68d0dc60-dd05-494b-b341-b97877d98ba4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:27.373923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.373904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d0dc60-dd05-494b-b341-b97877d98ba4-kube-api-access-7vldw" (OuterVolumeSpecName: "kube-api-access-7vldw") pod "68d0dc60-dd05-494b-b341-b97877d98ba4" (UID: "68d0dc60-dd05-494b-b341-b97877d98ba4"). InnerVolumeSpecName "kube-api-access-7vldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:27.374108 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.373924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68d0dc60-dd05-494b-b341-b97877d98ba4" (UID: "68d0dc60-dd05-494b-b341-b97877d98ba4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:27.472443 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.472336    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:27.472443 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.472376    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vldw\" (UniqueName: \"kubernetes.io/projected/68d0dc60-dd05-494b-b341-b97877d98ba4-kube-api-access-7vldw\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:27.472443 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.472391    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d0dc60-dd05-494b-b341-b97877d98ba4-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:27.472443 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:27.472403    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0dc60-dd05-494b-b341-b97877d98ba4-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.229046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.229009    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"
Sep 16 12:27:28.229464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.229005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b" event={"ID":"68d0dc60-dd05-494b-b341-b97877d98ba4","Type":"ContainerDied","Data":"97c42eaeb3ebfbe66e11da13bbe051f427533bc5c8d32e354da7ca68e8e6bb7b"}
Sep 16 12:27:28.229464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.229160    2578 scope.go:117] "RemoveContainer" containerID="6188aca2029ea8ac94b7c21214fe0cdbea6c8be031d9683369084d4cc6b3b138"
Sep 16 12:27:28.266097 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.266069    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"]
Sep 16 12:27:28.269353 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.269329    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-556077-6d58cfd9c4-bvx8b"]
Sep 16 12:27:28.308155 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.308103    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6" podUID="84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" containerName="main" containerID="cri-o://17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3" gracePeriod=45
Sep 16 12:27:28.315387 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.315358    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5" podUID="778a4350-37cd-43d1-bfc7-0a8084345aed" containerName="main" containerID="cri-o://421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b" gracePeriod=45
Sep 16 12:27:28.320929 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.320902    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j" podUID="3852cef5-bbbf-4f0e-a746-0f44164cebeb" containerName="main" containerID="cri-o://6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2" gracePeriod=45
Sep 16 12:27:28.342037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.341994    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn" podUID="1302f9e5-1121-47d1-9a5b-821fcc994388" containerName="main" containerID="cri-o://ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d" gracePeriod=45
Sep 16 12:27:28.669846 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.669717    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:27:28.717904 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.717878    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:27:28.722148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.722114    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:27:28.726137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.726117    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:27:28.787246 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-dshm\") pod \"778a4350-37cd-43d1-bfc7-0a8084345aed\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") "
Sep 16 12:27:28.787246 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-home\") pod \"1302f9e5-1121-47d1-9a5b-821fcc994388\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") "
Sep 16 12:27:28.787246 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td2vc\" (UniqueName: \"kubernetes.io/projected/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-kube-api-access-td2vc\") pod \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787267    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1302f9e5-1121-47d1-9a5b-821fcc994388-tls-certs\") pod \"1302f9e5-1121-47d1-9a5b-821fcc994388\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-home\") pod \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-model-cache\") pod \"1302f9e5-1121-47d1-9a5b-821fcc994388\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-model-cache\") pod \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-model-cache\") pod \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-tmp-dir\") pod \"778a4350-37cd-43d1-bfc7-0a8084345aed\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tmp-dir\") pod \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf4h7\" (UniqueName: \"kubernetes.io/projected/3852cef5-bbbf-4f0e-a746-0f44164cebeb-kube-api-access-jf4h7\") pod \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") "
Sep 16 12:27:28.787564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787541    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tmp-dir\") pod \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tls-certs\") pod \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787599    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a4350-37cd-43d1-bfc7-0a8084345aed-tls-certs\") pod \"778a4350-37cd-43d1-bfc7-0a8084345aed\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787636    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-dshm\") pod \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvfrj\" (UniqueName: \"kubernetes.io/projected/1302f9e5-1121-47d1-9a5b-821fcc994388-kube-api-access-nvfrj\") pod \"1302f9e5-1121-47d1-9a5b-821fcc994388\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-home\") pod \"778a4350-37cd-43d1-bfc7-0a8084345aed\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nscdt\" (UniqueName: \"kubernetes.io/projected/778a4350-37cd-43d1-bfc7-0a8084345aed-kube-api-access-nscdt\") pod \"778a4350-37cd-43d1-bfc7-0a8084345aed\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787818    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-tmp-dir\") pod \"1302f9e5-1121-47d1-9a5b-821fcc994388\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tls-certs\") pod \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\" (UID: \"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787901    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-dshm\") pod \"1302f9e5-1121-47d1-9a5b-821fcc994388\" (UID: \"1302f9e5-1121-47d1-9a5b-821fcc994388\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-dshm\") pod \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-home\") pod \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\" (UID: \"3852cef5-bbbf-4f0e-a746-0f44164cebeb\") "
Sep 16 12:27:28.788038 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.787996    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-model-cache\") pod \"778a4350-37cd-43d1-bfc7-0a8084345aed\" (UID: \"778a4350-37cd-43d1-bfc7-0a8084345aed\") "
Sep 16 12:27:28.788669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.788478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3852cef5-bbbf-4f0e-a746-0f44164cebeb" (UID: "3852cef5-bbbf-4f0e-a746-0f44164cebeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.788669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.788493    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-home" (OuterVolumeSpecName: "home") pod "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" (UID: "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.788669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.788572    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-home" (OuterVolumeSpecName: "home") pod "778a4350-37cd-43d1-bfc7-0a8084345aed" (UID: "778a4350-37cd-43d1-bfc7-0a8084345aed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.788827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.788755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-home" (OuterVolumeSpecName: "home") pod "3852cef5-bbbf-4f0e-a746-0f44164cebeb" (UID: "3852cef5-bbbf-4f0e-a746-0f44164cebeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.789280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.789195    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.789280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.789222    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.789280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.789236    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.789280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.789249    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.789600 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.789573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-model-cache" (OuterVolumeSpecName: "model-cache") pod "778a4350-37cd-43d1-bfc7-0a8084345aed" (UID: "778a4350-37cd-43d1-bfc7-0a8084345aed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.789811 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.789789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-home" (OuterVolumeSpecName: "home") pod "1302f9e5-1121-47d1-9a5b-821fcc994388" (UID: "1302f9e5-1121-47d1-9a5b-821fcc994388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.790059 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.790032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-model-cache" (OuterVolumeSpecName: "model-cache") pod "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" (UID: "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.790273 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.790251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-model-cache" (OuterVolumeSpecName: "model-cache") pod "1302f9e5-1121-47d1-9a5b-821fcc994388" (UID: "1302f9e5-1121-47d1-9a5b-821fcc994388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.790427 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.790374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-dshm" (OuterVolumeSpecName: "dshm") pod "778a4350-37cd-43d1-bfc7-0a8084345aed" (UID: "778a4350-37cd-43d1-bfc7-0a8084345aed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.790496 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.790442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-model-cache" (OuterVolumeSpecName: "model-cache") pod "3852cef5-bbbf-4f0e-a746-0f44164cebeb" (UID: "3852cef5-bbbf-4f0e-a746-0f44164cebeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.790661 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.790636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1302f9e5-1121-47d1-9a5b-821fcc994388" (UID: "1302f9e5-1121-47d1-9a5b-821fcc994388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.791690 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.791666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "778a4350-37cd-43d1-bfc7-0a8084345aed" (UID: "778a4350-37cd-43d1-bfc7-0a8084345aed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.791886 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.791863    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" (UID: "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.792481 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.792380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-dshm" (OuterVolumeSpecName: "dshm") pod "3852cef5-bbbf-4f0e-a746-0f44164cebeb" (UID: "3852cef5-bbbf-4f0e-a746-0f44164cebeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.793148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.793126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" (UID: "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:28.793278 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.793129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-dshm" (OuterVolumeSpecName: "dshm") pod "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" (UID: "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.793475 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.793439    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3852cef5-bbbf-4f0e-a746-0f44164cebeb" (UID: "3852cef5-bbbf-4f0e-a746-0f44164cebeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:28.793774 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.793741    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1302f9e5-1121-47d1-9a5b-821fcc994388-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1302f9e5-1121-47d1-9a5b-821fcc994388" (UID: "1302f9e5-1121-47d1-9a5b-821fcc994388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:28.794240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.794068    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-kube-api-access-td2vc" (OuterVolumeSpecName: "kube-api-access-td2vc") pod "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" (UID: "84800a49-1cb0-47c4-ace3-d9e6b5e4cda4"). InnerVolumeSpecName "kube-api-access-td2vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:28.794562 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.794400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778a4350-37cd-43d1-bfc7-0a8084345aed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "778a4350-37cd-43d1-bfc7-0a8084345aed" (UID: "778a4350-37cd-43d1-bfc7-0a8084345aed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:28.794562 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.794412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-dshm" (OuterVolumeSpecName: "dshm") pod "1302f9e5-1121-47d1-9a5b-821fcc994388" (UID: "1302f9e5-1121-47d1-9a5b-821fcc994388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:28.794562 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.794420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778a4350-37cd-43d1-bfc7-0a8084345aed-kube-api-access-nscdt" (OuterVolumeSpecName: "kube-api-access-nscdt") pod "778a4350-37cd-43d1-bfc7-0a8084345aed" (UID: "778a4350-37cd-43d1-bfc7-0a8084345aed"). InnerVolumeSpecName "kube-api-access-nscdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:28.794562 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.794492    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1302f9e5-1121-47d1-9a5b-821fcc994388-kube-api-access-nvfrj" (OuterVolumeSpecName: "kube-api-access-nvfrj") pod "1302f9e5-1121-47d1-9a5b-821fcc994388" (UID: "1302f9e5-1121-47d1-9a5b-821fcc994388"). InnerVolumeSpecName "kube-api-access-nvfrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:28.795317 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.795287    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3852cef5-bbbf-4f0e-a746-0f44164cebeb-kube-api-access-jf4h7" (OuterVolumeSpecName: "kube-api-access-jf4h7") pod "3852cef5-bbbf-4f0e-a746-0f44164cebeb" (UID: "3852cef5-bbbf-4f0e-a746-0f44164cebeb"). InnerVolumeSpecName "kube-api-access-jf4h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:28.890540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890482    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfrj\" (UniqueName: \"kubernetes.io/projected/1302f9e5-1121-47d1-9a5b-821fcc994388-kube-api-access-nvfrj\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890546    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscdt\" (UniqueName: \"kubernetes.io/projected/778a4350-37cd-43d1-bfc7-0a8084345aed-kube-api-access-nscdt\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890561    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890575    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890586    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890596    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890607    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890618    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890629    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890640    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td2vc\" (UniqueName: \"kubernetes.io/projected/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-kube-api-access-td2vc\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890641    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d0dc60-dd05-494b-b341-b97877d98ba4" path="/var/lib/kubelet/pods/68d0dc60-dd05-494b-b341-b97877d98ba4/volumes"
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890651    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1302f9e5-1121-47d1-9a5b-821fcc994388-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.890695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890692    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1302f9e5-1121-47d1-9a5b-821fcc994388-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.891081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890703    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3852cef5-bbbf-4f0e-a746-0f44164cebeb-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.891081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890712    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.891081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890720    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a4350-37cd-43d1-bfc7-0a8084345aed-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.891081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890728    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.891081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890736    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4h7\" (UniqueName: \"kubernetes.io/projected/3852cef5-bbbf-4f0e-a746-0f44164cebeb-kube-api-access-jf4h7\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.891081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890745    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3852cef5-bbbf-4f0e-a746-0f44164cebeb-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.891081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890754    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a4350-37cd-43d1-bfc7-0a8084345aed-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:28.891081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:28.890761    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:29.240107 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.240074    2578 generic.go:356] "Generic (PLEG): container finished" podID="3852cef5-bbbf-4f0e-a746-0f44164cebeb" containerID="6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2" exitCode=0
Sep 16 12:27:29.240783 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.240274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j" event={"ID":"3852cef5-bbbf-4f0e-a746-0f44164cebeb","Type":"ContainerDied","Data":"6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2"}
Sep 16 12:27:29.241674 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.240385    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"
Sep 16 12:27:29.241674 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.241168    2578 scope.go:117] "RemoveContainer" containerID="6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2"
Sep 16 12:27:29.242243 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.241003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j" event={"ID":"3852cef5-bbbf-4f0e-a746-0f44164cebeb","Type":"ContainerDied","Data":"5c20798b5aaa5ea937d3857bb884b836a5bbcae32c5541d3a22397ad7f0e87e4"}
Sep 16 12:27:29.244252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.243981    2578 generic.go:356] "Generic (PLEG): container finished" podID="1302f9e5-1121-47d1-9a5b-821fcc994388" containerID="ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d" exitCode=0
Sep 16 12:27:29.244252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.244054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn" event={"ID":"1302f9e5-1121-47d1-9a5b-821fcc994388","Type":"ContainerDied","Data":"ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d"}
Sep 16 12:27:29.244252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.244082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn" event={"ID":"1302f9e5-1121-47d1-9a5b-821fcc994388","Type":"ContainerDied","Data":"e8799c14bfc37abec18c2fe9b76ec945080065b535eae492e92c23a57ab7ccf2"}
Sep 16 12:27:29.244252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.244163    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"
Sep 16 12:27:29.251164 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.251125    2578 generic.go:356] "Generic (PLEG): container finished" podID="84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" containerID="17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3" exitCode=0
Sep 16 12:27:29.251485 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.251474    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"
Sep 16 12:27:29.251737 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.251720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6" event={"ID":"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4","Type":"ContainerDied","Data":"17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3"}
Sep 16 12:27:29.251893 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.251873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6" event={"ID":"84800a49-1cb0-47c4-ace3-d9e6b5e4cda4","Type":"ContainerDied","Data":"974bafbd04e0faa90ef43c0455e811a100c1bd3f12d14ad163de0e22d3453235"}
Sep 16 12:27:29.261559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.261496    2578 generic.go:356] "Generic (PLEG): container finished" podID="778a4350-37cd-43d1-bfc7-0a8084345aed" containerID="421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b" exitCode=0
Sep 16 12:27:29.261650 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.261565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5" event={"ID":"778a4350-37cd-43d1-bfc7-0a8084345aed","Type":"ContainerDied","Data":"421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b"}
Sep 16 12:27:29.261650 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.261573    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"
Sep 16 12:27:29.261728 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.261612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5" event={"ID":"778a4350-37cd-43d1-bfc7-0a8084345aed","Type":"ContainerDied","Data":"1a8b7e56cae7c036b0a64b0ef0fbabd558234e8fac27043eef806635ac59bc54"}
Sep 16 12:27:29.264448 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.264431    2578 scope.go:117] "RemoveContainer" containerID="6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2"
Sep 16 12:27:29.264770 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:29.264749    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2\": container with ID starting with 6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2 not found: ID does not exist" containerID="6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2"
Sep 16 12:27:29.264864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.264780    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2"} err="failed to get container status \"6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2\": rpc error: code = NotFound desc = could not find container \"6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2\": container with ID starting with 6cf919adabed62ae7222ea8915022c4efe146eccc84a1c705139de1f825ffbd2 not found: ID does not exist"
Sep 16 12:27:29.264864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.264802    2578 scope.go:117] "RemoveContainer" containerID="ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d"
Sep 16 12:27:29.273698 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.273677    2578 scope.go:117] "RemoveContainer" containerID="ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d"
Sep 16 12:27:29.273977 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:29.273956    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d\": container with ID starting with ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d not found: ID does not exist" containerID="ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d"
Sep 16 12:27:29.274057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.273983    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d"} err="failed to get container status \"ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d\": rpc error: code = NotFound desc = could not find container \"ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d\": container with ID starting with ef270a3ccc41461a24ef235a81ee4d71525b4b90488675d9c8dd62c812680a2d not found: ID does not exist"
Sep 16 12:27:29.274057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.273999    2578 scope.go:117] "RemoveContainer" containerID="17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3"
Sep 16 12:27:29.274328 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.274311    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"]
Sep 16 12:27:29.281367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.281343    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-556077/premium-w4-556077-kserve-66fbf947b6-x5j9j"]
Sep 16 12:27:29.283461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.283447    2578 scope.go:117] "RemoveContainer" containerID="17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3"
Sep 16 12:27:29.283785 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:29.283766    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3\": container with ID starting with 17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3 not found: ID does not exist" containerID="17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3"
Sep 16 12:27:29.283870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.283793    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3"} err="failed to get container status \"17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3\": rpc error: code = NotFound desc = could not find container \"17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3\": container with ID starting with 17768a74c5a5f1d29eeba7d1d8ac04d46e7c6dcfededae475b382634bf6ecab3 not found: ID does not exist"
Sep 16 12:27:29.283870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.283815    2578 scope.go:117] "RemoveContainer" containerID="421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b"
Sep 16 12:27:29.290740 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.290719    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"]
Sep 16 12:27:29.292690 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.292672    2578 scope.go:117] "RemoveContainer" containerID="421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b"
Sep 16 12:27:29.292951 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:29.292931    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b\": container with ID starting with 421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b not found: ID does not exist" containerID="421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b"
Sep 16 12:27:29.293017 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.292956    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b"} err="failed to get container status \"421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b\": rpc error: code = NotFound desc = could not find container \"421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b\": container with ID starting with 421ee371f54472533ec3a03b9110d4c00b5e3639c34b922ace6bb3858b6fac3b not found: ID does not exist"
Sep 16 12:27:29.294433 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.294411    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-556077/simulator-w4-556077-kserve-54d58c7d75-vqxqn"]
Sep 16 12:27:29.304356 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.304330    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"]
Sep 16 12:27:29.306651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.306633    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-w4-556077-kserve-678b85cc45-pjtw5"]
Sep 16 12:27:29.316594 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.316571    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"]
Sep 16 12:27:29.320199 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:29.320180    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-556077/distinct-2-w4-556077-kserve-59b6674bd6-22mq6"]
Sep 16 12:27:30.886223 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:30.886187    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1302f9e5-1121-47d1-9a5b-821fcc994388" path="/var/lib/kubelet/pods/1302f9e5-1121-47d1-9a5b-821fcc994388/volumes"
Sep 16 12:27:30.886681 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:30.886597    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3852cef5-bbbf-4f0e-a746-0f44164cebeb" path="/var/lib/kubelet/pods/3852cef5-bbbf-4f0e-a746-0f44164cebeb/volumes"
Sep 16 12:27:30.886958 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:30.886942    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778a4350-37cd-43d1-bfc7-0a8084345aed" path="/var/lib/kubelet/pods/778a4350-37cd-43d1-bfc7-0a8084345aed/volumes"
Sep 16 12:27:30.887624 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:30.887605    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" path="/var/lib/kubelet/pods/84800a49-1cb0-47c4-ace3-d9e6b5e4cda4/volumes"
Sep 16 12:27:34.300394 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:34.300290    2578 generic.go:356] "Generic (PLEG): container finished" podID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerID="698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886" exitCode=0
Sep 16 12:27:34.300394 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:34.300366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" event={"ID":"264cb7cc-88e8-4f3b-bb28-28eea673481d","Type":"ContainerDied","Data":"698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886"}
Sep 16 12:27:34.301004 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:34.300990    2578 scope.go:117] "RemoveContainer" containerID="698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886"
Sep 16 12:27:34.302589 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:34.302550    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:27:34.302691 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:34.302653    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:27:34.302731 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:34.302700    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:27:35.307946 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:35.307912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" event={"ID":"264cb7cc-88e8-4f3b-bb28-28eea673481d","Type":"ContainerStarted","Data":"df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356"}
Sep 16 12:27:36.107727 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:36.107690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:27:36.108060 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:36.108028    2578 patch_prober.go:28] interesting pod/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:27:36.108184 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:36.108089    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:27:37.007110 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:37.007067    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"]
Sep 16 12:27:37.045129 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:37.045081    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"]
Sep 16 12:27:37.108294 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:37.108253    2578 patch_prober.go:28] interesting pod/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:27:37.108505 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:37.108336    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:27:38.108212 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:38.108170    2578 patch_prober.go:28] interesting pod/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:27:38.108649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:38.108233    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:27:39.107485 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:39.107456    2578 patch_prober.go:28] interesting pod/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:27:39.107684 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:39.107506    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:27:40.107304 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:40.107265    2578 patch_prober.go:28] interesting pod/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 16 12:27:40.107733 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:40.107323    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 16 12:27:41.111667 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:41.111639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:27:41.112060 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:41.111898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:27:41.112620 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:41.112604    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:27:45.909754 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:45.909718    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"]
Sep 16 12:27:45.910230 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:45.909976    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" containerID="cri-o://9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6" gracePeriod=30
Sep 16 12:27:47.982607 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:47.982569    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"]
Sep 16 12:27:47.983067 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:47.982838    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh" podUID="69b5b387-09f6-415e-bba9-902c25a43bb9" containerName="maas-api" containerID="cri-o://a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616" gracePeriod=30
Sep 16 12:27:48.240762 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.240701    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:27:48.387487 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.387203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b5b387-09f6-415e-bba9-902c25a43bb9-openshift-service-ca\") pod \"69b5b387-09f6-415e-bba9-902c25a43bb9\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") "
Sep 16 12:27:48.387487 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.387272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls\") pod \"69b5b387-09f6-415e-bba9-902c25a43bb9\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") "
Sep 16 12:27:48.387487 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.387312    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87kpm\" (UniqueName: \"kubernetes.io/projected/69b5b387-09f6-415e-bba9-902c25a43bb9-kube-api-access-87kpm\") pod \"69b5b387-09f6-415e-bba9-902c25a43bb9\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") "
Sep 16 12:27:48.387487 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.387357    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-metrics-certs\") pod \"69b5b387-09f6-415e-bba9-902c25a43bb9\" (UID: \"69b5b387-09f6-415e-bba9-902c25a43bb9\") "
Sep 16 12:27:48.388215 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.388170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b5b387-09f6-415e-bba9-902c25a43bb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69b5b387-09f6-415e-bba9-902c25a43bb9" (UID: "69b5b387-09f6-415e-bba9-902c25a43bb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:48.389879 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.388305    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b5b387-09f6-415e-bba9-902c25a43bb9-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:48.393199 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.393145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b5b387-09f6-415e-bba9-902c25a43bb9-kube-api-access-87kpm" (OuterVolumeSpecName: "kube-api-access-87kpm") pod "69b5b387-09f6-415e-bba9-902c25a43bb9" (UID: "69b5b387-09f6-415e-bba9-902c25a43bb9"). InnerVolumeSpecName "kube-api-access-87kpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:48.396479 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.394729    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69b5b387-09f6-415e-bba9-902c25a43bb9" (UID: "69b5b387-09f6-415e-bba9-902c25a43bb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:48.396479 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.395181    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "69b5b387-09f6-415e-bba9-902c25a43bb9" (UID: "69b5b387-09f6-415e-bba9-902c25a43bb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:48.397682 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.397655    2578 generic.go:356] "Generic (PLEG): container finished" podID="69b5b387-09f6-415e-bba9-902c25a43bb9" containerID="a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616" exitCode=0
Sep 16 12:27:48.397803 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.397786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh" event={"ID":"69b5b387-09f6-415e-bba9-902c25a43bb9","Type":"ContainerDied","Data":"a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616"}
Sep 16 12:27:48.397868 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.397838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh" event={"ID":"69b5b387-09f6-415e-bba9-902c25a43bb9","Type":"ContainerDied","Data":"d9f5d67c28aa4421c4b5023146fd0b1faf7d327e4f3e5957987e75238bbaf3d6"}
Sep 16 12:27:48.397868 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.397865    2578 scope.go:117] "RemoveContainer" containerID="a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616"
Sep 16 12:27:48.398048 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.398034    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"
Sep 16 12:27:48.419087 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.419063    2578 scope.go:117] "RemoveContainer" containerID="a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616"
Sep 16 12:27:48.419419 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:48.419397    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616\": container with ID starting with a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616 not found: ID does not exist" containerID="a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616"
Sep 16 12:27:48.419482 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.419429    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616"} err="failed to get container status \"a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616\": rpc error: code = NotFound desc = could not find container \"a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616\": container with ID starting with a3acc52dd125651675d4163e329a74ea5065fd22d01cd8883b95a1b45902f616 not found: ID does not exist"
Sep 16 12:27:48.431065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.431035    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"]
Sep 16 12:27:48.437416 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.437387    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ff057a-6968b45bcc-fmdqh"]
Sep 16 12:27:48.489390 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.489354    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:48.489390 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.489386    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87kpm\" (UniqueName: \"kubernetes.io/projected/69b5b387-09f6-415e-bba9-902c25a43bb9-kube-api-access-87kpm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:48.489626 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.489404    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b5b387-09f6-415e-bba9-902c25a43bb9-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:48.885159 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:48.885126    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b5b387-09f6-415e-bba9-902c25a43bb9" path="/var/lib/kubelet/pods/69b5b387-09f6-415e-bba9-902c25a43bb9/volumes"
Sep 16 12:27:49.132910 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:49.132873    2578 patch_prober.go:28] interesting pod/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 16 12:27:49.133313 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:49.132950    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:27:51.176547 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.176465    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:27:51.320395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320352    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-socket\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.320395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320400    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-credential-socket\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.320702 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phxt2\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-kube-api-access-phxt2\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.320702 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-envoy\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.320702 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-certs\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.320702 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320611    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2fe014-795f-490e-8920-a7f8a9e9889d-istiod-ca-cert\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.320702 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-token\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.320702 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-data\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.321029 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:51.321029 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:51.321029 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.320936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:51.321207 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.321027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:51.321207 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.321077    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2fe014-795f-490e-8920-a7f8a9e9889d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:51.321207 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.321101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-podinfo\") pod \"3e2fe014-795f-490e-8920-a7f8a9e9889d\" (UID: \"3e2fe014-795f-490e-8920-a7f8a9e9889d\") "
Sep 16 12:27:51.321490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.321466    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-data\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.321490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.321492    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.321669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.321507    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-credential-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.321669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.321541    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-workload-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.321669 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.321554    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2fe014-795f-490e-8920-a7f8a9e9889d-istiod-ca-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.323154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.323130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:51.323280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.323249    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:27:51.323345 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.323326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-kube-api-access-phxt2" (OuterVolumeSpecName: "kube-api-access-phxt2") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "kube-api-access-phxt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:51.323345 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.323341    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e2fe014-795f-490e-8920-a7f8a9e9889d" (UID: "3e2fe014-795f-490e-8920-a7f8a9e9889d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:51.417592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.417560    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerID="9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6" exitCode=0
Sep 16 12:27:51.417864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.417615    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"
Sep 16 12:27:51.417864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.417650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" event={"ID":"3e2fe014-795f-490e-8920-a7f8a9e9889d","Type":"ContainerDied","Data":"9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6"}
Sep 16 12:27:51.417864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.417718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv" event={"ID":"3e2fe014-795f-490e-8920-a7f8a9e9889d","Type":"ContainerDied","Data":"03381e1e01b9db3f5c7213d0fe52156dc5256beb776093e49fe85df58f7331e6"}
Sep 16 12:27:51.417864 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.417753    2578 scope.go:117] "RemoveContainer" containerID="9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6"
Sep 16 12:27:51.423147 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.423118    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxt2\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-kube-api-access-phxt2\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.423260 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.423152    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-envoy\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.423260 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.423166    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-token\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.423260 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.423178    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2fe014-795f-490e-8920-a7f8a9e9889d-istio-podinfo\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:51.428850 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.428836    2578 scope.go:117] "RemoveContainer" containerID="95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64"
Sep 16 12:27:51.437933 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.437911    2578 scope.go:117] "RemoveContainer" containerID="9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6"
Sep 16 12:27:51.438249 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:51.438230    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6\": container with ID starting with 9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6 not found: ID does not exist" containerID="9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6"
Sep 16 12:27:51.438297 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.438258    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6"} err="failed to get container status \"9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6\": rpc error: code = NotFound desc = could not find container \"9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6\": container with ID starting with 9fc936637373d502ed0f38c5f0f6cd9003f641c14ecd042c181776f9e4351fa6 not found: ID does not exist"
Sep 16 12:27:51.438297 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.438280    2578 scope.go:117] "RemoveContainer" containerID="95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64"
Sep 16 12:27:51.438593 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:51.438573    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64\": container with ID starting with 95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64 not found: ID does not exist" containerID="95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64"
Sep 16 12:27:51.438666 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.438603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64"} err="failed to get container status \"95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64\": rpc error: code = NotFound desc = could not find container \"95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64\": container with ID starting with 95bda03248ce91cc5def6d1478a75ad5aa4f0db78554142f342ab71108f07e64 not found: ID does not exist"
Sep 16 12:27:51.443838 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.443814    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"]
Sep 16 12:27:51.446325 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:51.446292    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a6f57e-openshift-default-5586dc947b-z42nv"]
Sep 16 12:27:52.027441 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.027404    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww" podUID="0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" containerName="main" containerID="cri-o://4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df" gracePeriod=45
Sep 16 12:27:52.071167 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.071131    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp" podUID="03188b71-c35e-48cc-a837-6d024c966f4b" containerName="main" containerID="cri-o://24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095" gracePeriod=45
Sep 16 12:27:52.343659 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.343633    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:27:52.347069 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.347037    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:27:52.425485 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.425453    2578 generic.go:356] "Generic (PLEG): container finished" podID="03188b71-c35e-48cc-a837-6d024c966f4b" containerID="24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095" exitCode=0
Sep 16 12:27:52.425675 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.425495    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"
Sep 16 12:27:52.425675 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.425544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp" event={"ID":"03188b71-c35e-48cc-a837-6d024c966f4b","Type":"ContainerDied","Data":"24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095"}
Sep 16 12:27:52.425675 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.425585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp" event={"ID":"03188b71-c35e-48cc-a837-6d024c966f4b","Type":"ContainerDied","Data":"ba185de0063f5ebf17a1782e74c157775d1d5a68d4d48be3828a2320f3ac941b"}
Sep 16 12:27:52.425675 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.425613    2578 scope.go:117] "RemoveContainer" containerID="24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095"
Sep 16 12:27:52.428282 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.428253    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" containerID="4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df" exitCode=0
Sep 16 12:27:52.428417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.428309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww" event={"ID":"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b","Type":"ContainerDied","Data":"4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df"}
Sep 16 12:27:52.428417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.428361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww" event={"ID":"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b","Type":"ContainerDied","Data":"d0d2c24b91d3c34f5103487cedca73899a9d5680a8c0818450025a11edd89947"}
Sep 16 12:27:52.428417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.428313    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"
Sep 16 12:27:52.432702 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.432675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpr8t\" (UniqueName: \"kubernetes.io/projected/03188b71-c35e-48cc-a837-6d024c966f4b-kube-api-access-dpr8t\") pod \"03188b71-c35e-48cc-a837-6d024c966f4b\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") "
Sep 16 12:27:52.432830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.432713    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-model-cache\") pod \"03188b71-c35e-48cc-a837-6d024c966f4b\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") "
Sep 16 12:27:52.432830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.432731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-home\") pod \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") "
Sep 16 12:27:52.433124 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.432838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-tmp-dir\") pod \"03188b71-c35e-48cc-a837-6d024c966f4b\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") "
Sep 16 12:27:52.433195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433157    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-dshm\") pod \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") "
Sep 16 12:27:52.433254 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.432976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-home" (OuterVolumeSpecName: "home") pod "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" (UID: "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.433254 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.432985    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "03188b71-c35e-48cc-a837-6d024c966f4b" (UID: "03188b71-c35e-48cc-a837-6d024c966f4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.433254 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03188b71-c35e-48cc-a837-6d024c966f4b" (UID: "03188b71-c35e-48cc-a837-6d024c966f4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.433254 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t9fz\" (UniqueName: \"kubernetes.io/projected/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-kube-api-access-6t9fz\") pod \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") "
Sep 16 12:27:52.433470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-dshm\") pod \"03188b71-c35e-48cc-a837-6d024c966f4b\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") "
Sep 16 12:27:52.433470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433312    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tls-certs\") pod \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") "
Sep 16 12:27:52.433470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433390    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-model-cache\") pod \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") "
Sep 16 12:27:52.433470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188b71-c35e-48cc-a837-6d024c966f4b-tls-certs\") pod \"03188b71-c35e-48cc-a837-6d024c966f4b\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") "
Sep 16 12:27:52.433708 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433476    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-home\") pod \"03188b71-c35e-48cc-a837-6d024c966f4b\" (UID: \"03188b71-c35e-48cc-a837-6d024c966f4b\") "
Sep 16 12:27:52.433708 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tmp-dir\") pod \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\" (UID: \"0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b\") "
Sep 16 12:27:52.433708 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" (UID: "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.433870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" (UID: "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.433948 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433933    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.434006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433955    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.434006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433969    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.434006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433982    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.434006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.433995    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.434210 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.434113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-home" (OuterVolumeSpecName: "home") pod "03188b71-c35e-48cc-a837-6d024c966f4b" (UID: "03188b71-c35e-48cc-a837-6d024c966f4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.436150 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.436117    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03188b71-c35e-48cc-a837-6d024c966f4b-kube-api-access-dpr8t" (OuterVolumeSpecName: "kube-api-access-dpr8t") pod "03188b71-c35e-48cc-a837-6d024c966f4b" (UID: "03188b71-c35e-48cc-a837-6d024c966f4b"). InnerVolumeSpecName "kube-api-access-dpr8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:52.436281 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.436207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-dshm" (OuterVolumeSpecName: "dshm") pod "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" (UID: "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.436559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.436504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-dshm" (OuterVolumeSpecName: "dshm") pod "03188b71-c35e-48cc-a837-6d024c966f4b" (UID: "03188b71-c35e-48cc-a837-6d024c966f4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.436683 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.436630    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" (UID: "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:52.436683 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.436660    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-kube-api-access-6t9fz" (OuterVolumeSpecName: "kube-api-access-6t9fz") pod "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" (UID: "0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b"). InnerVolumeSpecName "kube-api-access-6t9fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:52.437152 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.437133    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03188b71-c35e-48cc-a837-6d024c966f4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03188b71-c35e-48cc-a837-6d024c966f4b" (UID: "03188b71-c35e-48cc-a837-6d024c966f4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:52.437471 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.437447    2578 scope.go:117] "RemoveContainer" containerID="24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095"
Sep 16 12:27:52.437889 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:52.437868    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095\": container with ID starting with 24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095 not found: ID does not exist" containerID="24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095"
Sep 16 12:27:52.437984 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.437896    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095"} err="failed to get container status \"24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095\": rpc error: code = NotFound desc = could not find container \"24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095\": container with ID starting with 24bf90721b2110cfea0f24ac63b2ce013528f84405bec04b9dd6ef9f4cacd095 not found: ID does not exist"
Sep 16 12:27:52.437984 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.437916    2578 scope.go:117] "RemoveContainer" containerID="4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df"
Sep 16 12:27:52.447086 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.447063    2578 scope.go:117] "RemoveContainer" containerID="4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df"
Sep 16 12:27:52.447446 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:27:52.447419    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df\": container with ID starting with 4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df not found: ID does not exist" containerID="4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df"
Sep 16 12:27:52.447603 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.447454    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df"} err="failed to get container status \"4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df\": rpc error: code = NotFound desc = could not find container \"4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df\": container with ID starting with 4aa855f9f865eaea545105a0a26bb432084187bc1b694347ab381af6b7b107df not found: ID does not exist"
Sep 16 12:27:52.535367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.535328    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188b71-c35e-48cc-a837-6d024c966f4b-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.535367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.535363    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.535367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.535372    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpr8t\" (UniqueName: \"kubernetes.io/projected/03188b71-c35e-48cc-a837-6d024c966f4b-kube-api-access-dpr8t\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.535635 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.535380    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.535635 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.535391    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t9fz\" (UniqueName: \"kubernetes.io/projected/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-kube-api-access-6t9fz\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.535635 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.535403    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188b71-c35e-48cc-a837-6d024c966f4b-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.535635 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.535417    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.749248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.749215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"]
Sep 16 12:27:52.754649 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.754618    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/simulator-w0-ff057a-kserve-6f899fb98d-znrjp"]
Sep 16 12:27:52.765341 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.765304    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"]
Sep 16 12:27:52.768085 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.768062    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-w0-ff057a-kserve-85f4ffd84-qw2ww"]
Sep 16 12:27:52.885770 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.885736    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03188b71-c35e-48cc-a837-6d024c966f4b" path="/var/lib/kubelet/pods/03188b71-c35e-48cc-a837-6d024c966f4b/volumes"
Sep 16 12:27:52.886156 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.886143    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" path="/var/lib/kubelet/pods/0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b/volumes"
Sep 16 12:27:52.886525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:27:52.886497    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" path="/var/lib/kubelet/pods/3e2fe014-795f-490e-8920-a7f8a9e9889d/volumes"
Sep 16 12:28:01.937078 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:01.937038    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"]
Sep 16 12:28:01.937614 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:01.937256    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6" podUID="6bdda458-958c-4ea0-a2d9-6b4e71b665cf" containerName="maas-api" containerID="cri-o://fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17" gracePeriod=30
Sep 16 12:28:02.176849 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.176822    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:28:02.320501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.320451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-metrics-certs\") pod \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") "
Sep 16 12:28:02.320501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.320502    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r92x\" (UniqueName: \"kubernetes.io/projected/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-kube-api-access-7r92x\") pod \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") "
Sep 16 12:28:02.320757 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.320572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-openshift-service-ca\") pod \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") "
Sep 16 12:28:02.320757 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.320659    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-maas-api-tls\") pod \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\" (UID: \"6bdda458-958c-4ea0-a2d9-6b4e71b665cf\") "
Sep 16 12:28:02.320994 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.320967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6bdda458-958c-4ea0-a2d9-6b4e71b665cf" (UID: "6bdda458-958c-4ea0-a2d9-6b4e71b665cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:02.322912 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.322885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6bdda458-958c-4ea0-a2d9-6b4e71b665cf" (UID: "6bdda458-958c-4ea0-a2d9-6b4e71b665cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:02.323024 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.322904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6bdda458-958c-4ea0-a2d9-6b4e71b665cf" (UID: "6bdda458-958c-4ea0-a2d9-6b4e71b665cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:02.323065 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.323016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-kube-api-access-7r92x" (OuterVolumeSpecName: "kube-api-access-7r92x") pod "6bdda458-958c-4ea0-a2d9-6b4e71b665cf" (UID: "6bdda458-958c-4ea0-a2d9-6b4e71b665cf"). InnerVolumeSpecName "kube-api-access-7r92x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:02.421877 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.421840    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:02.421877 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.421871    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:02.421877 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.421882    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r92x\" (UniqueName: \"kubernetes.io/projected/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-kube-api-access-7r92x\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:02.422113 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.421892    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdda458-958c-4ea0-a2d9-6b4e71b665cf-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:02.485677 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.485641    2578 generic.go:356] "Generic (PLEG): container finished" podID="6bdda458-958c-4ea0-a2d9-6b4e71b665cf" containerID="fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17" exitCode=0
Sep 16 12:28:02.485831 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.485694    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"
Sep 16 12:28:02.485831 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.485722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6" event={"ID":"6bdda458-958c-4ea0-a2d9-6b4e71b665cf","Type":"ContainerDied","Data":"fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17"}
Sep 16 12:28:02.485831 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.485780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6" event={"ID":"6bdda458-958c-4ea0-a2d9-6b4e71b665cf","Type":"ContainerDied","Data":"68493b44d2a3117968a5856c8a2d10acad9ba6a881d85023720591497f791689"}
Sep 16 12:28:02.485831 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.485803    2578 scope.go:117] "RemoveContainer" containerID="fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17"
Sep 16 12:28:02.495923 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.495907    2578 scope.go:117] "RemoveContainer" containerID="fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17"
Sep 16 12:28:02.496175 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:02.496155    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17\": container with ID starting with fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17 not found: ID does not exist" containerID="fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17"
Sep 16 12:28:02.496239 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.496187    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17"} err="failed to get container status \"fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17\": rpc error: code = NotFound desc = could not find container \"fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17\": container with ID starting with fc57283783afe2d9eda6a2f7863ba8912caf02f23c61e19adc8f19bd981fac17 not found: ID does not exist"
Sep 16 12:28:02.523240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.523217    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"]
Sep 16 12:28:02.538032 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.538001    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f927-649b66d8b-sp2r6"]
Sep 16 12:28:02.884621 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:02.884580    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bdda458-958c-4ea0-a2d9-6b4e71b665cf" path="/var/lib/kubelet/pods/6bdda458-958c-4ea0-a2d9-6b4e71b665cf/volumes"
Sep 16 12:28:08.652577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:08.652544    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"]
Sep 16 12:28:08.653053 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:08.652764    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy" containerID="cri-o://df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356" gracePeriod=30
Sep 16 12:28:11.114289 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:11.114252    2578 patch_prober.go:28] interesting pod/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:28:11.114724 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:11.114319    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:28:13.915776 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:13.915711    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:28:14.023495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264cb7cc-88e8-4f3b-bb28-28eea673481d-istiod-ca-cert\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.023712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-data\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.023712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023593    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-socket\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.023712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023636    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-podinfo\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.023712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-credential-socket\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.023712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-envoy\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.023712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw97x\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-kube-api-access-hw97x\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.023712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-token\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.024071 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-certs\") pod \"264cb7cc-88e8-4f3b-bb28-28eea673481d\" (UID: \"264cb7cc-88e8-4f3b-bb28-28eea673481d\") "
Sep 16 12:28:14.024071 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-data" (OuterVolumeSpecName: "istio-data") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:14.024071 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.023973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:14.024071 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.024009    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264cb7cc-88e8-4f3b-bb28-28eea673481d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:14.024288 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.024171    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:14.024288 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.024165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:14.024288 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.024275    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.024288 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.024288    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-credential-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.024493 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.024297    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-workload-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.024493 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.024306    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264cb7cc-88e8-4f3b-bb28-28eea673481d-istiod-ca-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.024493 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.024314    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-data\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.026123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.026087    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:14.026236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.026116    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-kube-api-access-hw97x" (OuterVolumeSpecName: "kube-api-access-hw97x") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "kube-api-access-hw97x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:14.026236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.026170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:28:14.026236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.026173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-token" (OuterVolumeSpecName: "istio-token") pod "264cb7cc-88e8-4f3b-bb28-28eea673481d" (UID: "264cb7cc-88e8-4f3b-bb28-28eea673481d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:14.125014 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.124971    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-podinfo\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.125014 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.125004    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-envoy\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.125014 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.125014    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw97x\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-kube-api-access-hw97x\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.125014 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.125024    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264cb7cc-88e8-4f3b-bb28-28eea673481d-istio-token\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:14.561396 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.561361    2578 generic.go:356] "Generic (PLEG): container finished" podID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerID="df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356" exitCode=0
Sep 16 12:28:14.561631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.561420    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"
Sep 16 12:28:14.561631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.561454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" event={"ID":"264cb7cc-88e8-4f3b-bb28-28eea673481d","Type":"ContainerDied","Data":"df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356"}
Sep 16 12:28:14.561631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.561586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv" event={"ID":"264cb7cc-88e8-4f3b-bb28-28eea673481d","Type":"ContainerDied","Data":"87a0304b7040123bd1e935f4f68bc49b07679383e23bf3add70400b097d6ab86"}
Sep 16 12:28:14.561631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.561620    2578 scope.go:117] "RemoveContainer" containerID="df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356"
Sep 16 12:28:14.574823 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.574799    2578 scope.go:117] "RemoveContainer" containerID="698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886"
Sep 16 12:28:14.584453 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.584432    2578 scope.go:117] "RemoveContainer" containerID="df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356"
Sep 16 12:28:14.584833 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:14.584808    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356\": container with ID starting with df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356 not found: ID does not exist" containerID="df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356"
Sep 16 12:28:14.584899 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.584844    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356"} err="failed to get container status \"df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356\": rpc error: code = NotFound desc = could not find container \"df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356\": container with ID starting with df8e1ecdef7a79741e40c63ca9e4757e8b416c3c52caf75e718d84fa59ddb356 not found: ID does not exist"
Sep 16 12:28:14.584899 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.584867    2578 scope.go:117] "RemoveContainer" containerID="698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886"
Sep 16 12:28:14.585201 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:14.585182    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886\": container with ID starting with 698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886 not found: ID does not exist" containerID="698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886"
Sep 16 12:28:14.585294 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.585205    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886"} err="failed to get container status \"698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886\": rpc error: code = NotFound desc = could not find container \"698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886\": container with ID starting with 698bb156f7a36bcd713da0385d1bef76dcd7ad33e2e33b237fe36511dbc13886 not found: ID does not exist"
Sep 16 12:28:14.588884 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.588859    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"]
Sep 16 12:28:14.592402 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.592373    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-30f927-openshift-default-5df44c9cc6-dwbrv"]
Sep 16 12:28:14.885353 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:14.885273    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" path="/var/lib/kubelet/pods/264cb7cc-88e8-4f3b-bb28-28eea673481d/volumes"
Sep 16 12:28:34.766255 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766223    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-226ed7"]
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766596    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778a4350-37cd-43d1-bfc7-0a8084345aed" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766608    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="778a4350-37cd-43d1-bfc7-0a8084345aed" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766618    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766624    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766635    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d0dc60-dd05-494b-b341-b97877d98ba4" containerName="maas-api"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766640    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d0dc60-dd05-494b-b341-b97877d98ba4" containerName="maas-api"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766646    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b5b387-09f6-415e-bba9-902c25a43bb9" containerName="maas-api"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766651    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b5b387-09f6-415e-bba9-902c25a43bb9" containerName="maas-api"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766660    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766665    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766672    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766677    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766684    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1302f9e5-1121-47d1-9a5b-821fcc994388" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766690    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1302f9e5-1121-47d1-9a5b-821fcc994388" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766696    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3852cef5-bbbf-4f0e-a746-0f44164cebeb" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3852cef5-bbbf-4f0e-a746-0f44164cebeb" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766708    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766713    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766721    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bdda458-958c-4ea0-a2d9-6b4e71b665cf" containerName="maas-api"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766726    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bdda458-958c-4ea0-a2d9-6b4e71b665cf" containerName="maas-api"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766732    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03188b71-c35e-48cc-a837-6d024c966f4b" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766738    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03188b71-c35e-48cc-a837-6d024c966f4b" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766745    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" containerName="main"
Sep 16 12:28:34.766759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766750    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" containerName="main"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766805    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1302f9e5-1121-47d1-9a5b-821fcc994388" containerName="main"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766813    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3852cef5-bbbf-4f0e-a746-0f44164cebeb" containerName="main"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766818    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d0dc60-dd05-494b-b341-b97877d98ba4" containerName="maas-api"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766823    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bdda458-958c-4ea0-a2d9-6b4e71b665cf" containerName="maas-api"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766829    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766833    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766838    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03188b71-c35e-48cc-a837-6d024c966f4b" containerName="main"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766844    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b5b387-09f6-415e-bba9-902c25a43bb9" containerName="maas-api"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766849    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d631bc2-f9e2-4c68-b7a9-266ff1b3c85b" containerName="main"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766854    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="84800a49-1cb0-47c4-ace3-d9e6b5e4cda4" containerName="main"
Sep 16 12:28:34.767490 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.766860    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="778a4350-37cd-43d1-bfc7-0a8084345aed" containerName="main"
Sep 16 12:28:34.769116 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.769101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:34.771676 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.771651    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zpzr9\""
Sep 16 12:28:34.773797 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.773766    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-226ed7"]
Sep 16 12:28:34.915800 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:34.915767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289lk\" (UniqueName: \"kubernetes.io/projected/ceaa9a35-2565-42a7-9524-c1674c8c5b4d-kube-api-access-289lk\") pod \"test-curl-7478-226ed7\" (UID: \"ceaa9a35-2565-42a7-9524-c1674c8c5b4d\") " pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:35.017130 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:35.017052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289lk\" (UniqueName: \"kubernetes.io/projected/ceaa9a35-2565-42a7-9524-c1674c8c5b4d-kube-api-access-289lk\") pod \"test-curl-7478-226ed7\" (UID: \"ceaa9a35-2565-42a7-9524-c1674c8c5b4d\") " pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:35.026908 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:35.026882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-289lk\" (UniqueName: \"kubernetes.io/projected/ceaa9a35-2565-42a7-9524-c1674c8c5b4d-kube-api-access-289lk\") pod \"test-curl-7478-226ed7\" (UID: \"ceaa9a35-2565-42a7-9524-c1674c8c5b4d\") " pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:35.079898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:35.079854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:35.352853 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:35.352822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-226ed7"]
Sep 16 12:28:35.353414 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:28:35.353386    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceaa9a35_2565_42a7_9524_c1674c8c5b4d.slice/crio-dddc930cda33fe15c4dd31e1374c6de66a9143272a5fc7725baaf5c834c6be0c WatchSource:0}: Error finding container dddc930cda33fe15c4dd31e1374c6de66a9143272a5fc7725baaf5c834c6be0c: Status 404 returned error can't find the container with id dddc930cda33fe15c4dd31e1374c6de66a9143272a5fc7725baaf5c834c6be0c
Sep 16 12:28:35.678699 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:35.678611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-226ed7" event={"ID":"ceaa9a35-2565-42a7-9524-c1674c8c5b4d","Type":"ContainerStarted","Data":"6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e"}
Sep 16 12:28:35.678699 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:35.678651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-226ed7" event={"ID":"ceaa9a35-2565-42a7-9524-c1674c8c5b4d","Type":"ContainerStarted","Data":"dddc930cda33fe15c4dd31e1374c6de66a9143272a5fc7725baaf5c834c6be0c"}
Sep 16 12:28:35.693006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:35.692960    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7478-226ed7" podStartSLOduration=1.692944755 podStartE2EDuration="1.692944755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:35.69221447 +0000 UTC m=+1605.329539308" watchObservedRunningTime="2026-09-16 12:28:35.692944755 +0000 UTC m=+1605.330269648"
Sep 16 12:28:36.312778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:36.312743    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-226ed7"]
Sep 16 12:28:36.315162 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:36.315137    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-226ed7"]
Sep 16 12:28:36.684208 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:36.684174    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7478-226ed7" container="test-curl-7478-226ed7"
Sep 16 12:28:37.069966 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.069935    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"]
Sep 16 12:28:37.070495 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.070453    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb" podUID="5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" containerName="maas-api" containerID="cri-o://5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a" gracePeriod=30
Sep 16 12:28:37.325629 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.325560    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:28:37.440689 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.440654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-metrics-certs\") pod \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") "
Sep 16 12:28:37.440893 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.440706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls\") pod \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") "
Sep 16 12:28:37.440893 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.440766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpb2l\" (UniqueName: \"kubernetes.io/projected/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-kube-api-access-mpb2l\") pod \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") "
Sep 16 12:28:37.440893 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.440835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-openshift-service-ca\") pod \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\" (UID: \"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678\") "
Sep 16 12:28:37.441462 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.441427    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" (UID: "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:37.443471 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.443436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-kube-api-access-mpb2l" (OuterVolumeSpecName: "kube-api-access-mpb2l") pod "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" (UID: "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678"). InnerVolumeSpecName "kube-api-access-mpb2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:37.443608 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.443563    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" (UID: "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:37.443687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.443654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" (UID: "5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:37.542587 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.542547    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:37.542587 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.542576    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:37.542587 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.542585    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpb2l\" (UniqueName: \"kubernetes.io/projected/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-kube-api-access-mpb2l\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:37.542821 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.542596    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:37.690501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.690418    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" containerID="5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a" exitCode=0
Sep 16 12:28:37.690501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.690483    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"
Sep 16 12:28:37.690687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.690504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb" event={"ID":"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678","Type":"ContainerDied","Data":"5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a"}
Sep 16 12:28:37.690687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.690615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb" event={"ID":"5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678","Type":"ContainerDied","Data":"763120e01cf8b3d2394a7f3a9042455ddb84e6e7fe1058c88cd767ac65eed4f7"}
Sep 16 12:28:37.690687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.690644    2578 scope.go:117] "RemoveContainer" containerID="5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a"
Sep 16 12:28:37.690807 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.690782    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7478-226ed7" podUID="ceaa9a35-2565-42a7-9524-c1674c8c5b4d" containerName="test-curl-7478-226ed7" containerID="cri-o://6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e" gracePeriod=2
Sep 16 12:28:37.700532 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.700501    2578 scope.go:117] "RemoveContainer" containerID="5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a"
Sep 16 12:28:37.700802 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:37.700783    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a\": container with ID starting with 5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a not found: ID does not exist" containerID="5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a"
Sep 16 12:28:37.700870 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.700808    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a"} err="failed to get container status \"5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a\": rpc error: code = NotFound desc = could not find container \"5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a\": container with ID starting with 5cccf00d39426509a4388e5ed707ba4a25c1730cf5f43d6324240840b55abe8a not found: ID does not exist"
Sep 16 12:28:37.719105 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.719080    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"]
Sep 16 12:28:37.725024 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:37.725004    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6f8610-58f7d4d784-v4dvb"]
Sep 16 12:28:38.885003 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:38.884971    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" path="/var/lib/kubelet/pods/5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678/volumes"
Sep 16 12:28:39.864003 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:39.863981    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:39.963292 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:39.963261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-289lk\" (UniqueName: \"kubernetes.io/projected/ceaa9a35-2565-42a7-9524-c1674c8c5b4d-kube-api-access-289lk\") pod \"ceaa9a35-2565-42a7-9524-c1674c8c5b4d\" (UID: \"ceaa9a35-2565-42a7-9524-c1674c8c5b4d\") "
Sep 16 12:28:39.965499 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:39.965475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaa9a35-2565-42a7-9524-c1674c8c5b4d-kube-api-access-289lk" (OuterVolumeSpecName: "kube-api-access-289lk") pod "ceaa9a35-2565-42a7-9524-c1674c8c5b4d" (UID: "ceaa9a35-2565-42a7-9524-c1674c8c5b4d"). InnerVolumeSpecName "kube-api-access-289lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:40.064007 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:40.063972    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-289lk\" (UniqueName: \"kubernetes.io/projected/ceaa9a35-2565-42a7-9524-c1674c8c5b4d-kube-api-access-289lk\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:40.708865 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:40.708831    2578 generic.go:356] "Generic (PLEG): container finished" podID="ceaa9a35-2565-42a7-9524-c1674c8c5b4d" containerID="6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e" exitCode=137
Sep 16 12:28:40.709096 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:40.708879    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:40.709096 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:40.708899    2578 scope.go:117] "RemoveContainer" containerID="6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e"
Sep 16 12:28:40.718016 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:40.718000    2578 scope.go:117] "RemoveContainer" containerID="6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e"
Sep 16 12:28:40.718284 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:40.718266    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e\": container with ID starting with 6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e not found: ID does not exist" containerID="6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e"
Sep 16 12:28:40.718342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:40.718298    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e"} err="failed to get container status \"6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e\": rpc error: code = NotFound desc = could not find container \"6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e\": container with ID starting with 6b4f39b4c612dfe489176dc545ec6efd7cbbae1831586b82c98afd27fd7c3d7e not found: ID does not exist"
Sep 16 12:28:40.719223 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:40.719201    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-226ed7\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" podUID="ceaa9a35-2565-42a7-9524-c1674c8c5b4d" pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:40.882990 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:40.882952    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-226ed7\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" podUID="ceaa9a35-2565-42a7-9524-c1674c8c5b4d" pod="opendatahub/test-curl-7478-226ed7"
Sep 16 12:28:40.883975 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:40.883954    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceaa9a35-2565-42a7-9524-c1674c8c5b4d" path="/var/lib/kubelet/pods/ceaa9a35-2565-42a7-9524-c1674c8c5b4d/volumes"
Sep 16 12:28:44.137110 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:44.137006    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"]
Sep 16 12:28:44.137563 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:44.137292    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy" containerID="cri-o://22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c" gracePeriod=30
Sep 16 12:28:46.855035 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.854998    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"]
Sep 16 12:28:46.855506 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855466    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceaa9a35-2565-42a7-9524-c1674c8c5b4d" containerName="test-curl-7478-226ed7"
Sep 16 12:28:46.855506 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855485    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceaa9a35-2565-42a7-9524-c1674c8c5b4d" containerName="test-curl-7478-226ed7"
Sep 16 12:28:46.855506 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855503    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" containerName="maas-api"
Sep 16 12:28:46.855704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855532    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" containerName="maas-api"
Sep 16 12:28:46.855704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855559    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy"
Sep 16 12:28:46.855704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855567    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy"
Sep 16 12:28:46.855704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855668    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6504b0-d41b-4d9a-ac8e-a3bcfb5dd678" containerName="maas-api"
Sep 16 12:28:46.855704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855680    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2fe014-795f-490e-8920-a7f8a9e9889d" containerName="istio-proxy"
Sep 16 12:28:46.855704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855689    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="264cb7cc-88e8-4f3b-bb28-28eea673481d" containerName="istio-proxy"
Sep 16 12:28:46.855704 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.855698    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceaa9a35-2565-42a7-9524-c1674c8c5b4d" containerName="test-curl-7478-226ed7"
Sep 16 12:28:46.858550 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.858527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:46.862398 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.862375    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-db42a6\"/\"openshift-service-ca.crt\""
Sep 16 12:28:46.862553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.862427    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-db42a6\"/\"kube-root-ca.crt\""
Sep 16 12:28:46.862553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.862458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db42a6\"/\"sub-test-model-db42a6-kserve-self-signed-certs\""
Sep 16 12:28:46.862553 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.862374    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db42a6\"/\"default-dockercfg-czqpt\""
Sep 16 12:28:46.867698 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:46.867670    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"]
Sep 16 12:28:47.021282 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.021252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-dshm\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.021479 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.021291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndf8\" (UniqueName: \"kubernetes.io/projected/5b04e141-d672-4198-af0a-7f83ad01b14a-kube-api-access-fndf8\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.021479 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.021314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-tmp-dir\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.021479 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.021331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-model-cache\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.021639 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.021469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04e141-d672-4198-af0a-7f83ad01b14a-tls-certs\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.021639 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.021541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-home\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.122577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.122438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-dshm\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.122577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.122499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndf8\" (UniqueName: \"kubernetes.io/projected/5b04e141-d672-4198-af0a-7f83ad01b14a-kube-api-access-fndf8\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.122577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.122548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-tmp-dir\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.122577 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.122564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-model-cache\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.122945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.122622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04e141-d672-4198-af0a-7f83ad01b14a-tls-certs\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.122945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.122663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-home\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.123020 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.122995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-model-cache\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.123061 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.123037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-tmp-dir\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.123109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.123062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-home\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.124973 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.124939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-dshm\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.125302 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.125283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04e141-d672-4198-af0a-7f83ad01b14a-tls-certs\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.130617 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.130595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndf8\" (UniqueName: \"kubernetes.io/projected/5b04e141-d672-4198-af0a-7f83ad01b14a-kube-api-access-fndf8\") pod \"sub-test-model-db42a6-kserve-59947cbb65-4hhbt\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.169880 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.169855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:47.452890 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.452863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"]
Sep 16 12:28:47.453708 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:28:47.453682    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b04e141_d672_4198_af0a_7f83ad01b14a.slice/crio-2486db84cef229275aefcced9bd015b5fdc3e13b000354f115a1fdfab765c364 WatchSource:0}: Error finding container 2486db84cef229275aefcced9bd015b5fdc3e13b000354f115a1fdfab765c364: Status 404 returned error can't find the container with id 2486db84cef229275aefcced9bd015b5fdc3e13b000354f115a1fdfab765c364
Sep 16 12:28:47.747464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.747428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt" event={"ID":"5b04e141-d672-4198-af0a-7f83ad01b14a","Type":"ContainerStarted","Data":"358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e"}
Sep 16 12:28:47.747635 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.747474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt" event={"ID":"5b04e141-d672-4198-af0a-7f83ad01b14a","Type":"ContainerStarted","Data":"2486db84cef229275aefcced9bd015b5fdc3e13b000354f115a1fdfab765c364"}
Sep 16 12:28:47.763906 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:47.763864    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt" podStartSLOduration=1.763848285 podStartE2EDuration="1.763848285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:47.76231962 +0000 UTC m=+1617.399644469" watchObservedRunningTime="2026-09-16 12:28:47.763848285 +0000 UTC m=+1617.401173123"
Sep 16 12:28:49.378332 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.378305    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:28:49.446546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-podinfo\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446546 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446498    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istiod-ca-cert\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-socket\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446668    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-data\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-credential-socket\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446725    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-envoy\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446968 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-token\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446968 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:49.446968 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-certs\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446968 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk7nw\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-kube-api-access-kk7nw\") pod \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\" (UID: \"ba6fc64a-c1a9-41dd-aa41-4f08230fd589\") "
Sep 16 12:28:49.446968 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446946    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:49.447206 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.446978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:49.447206 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.447030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:49.447206 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.447158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:49.447570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.447448    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.447570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.447474    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-data\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.447570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.447490    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-credential-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.447570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.447504    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-workload-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.447570 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.447547    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istiod-ca-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.448944 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.448921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:28:49.449069 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.449030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:49.449537 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.449499    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:49.449624 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.449534    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-kube-api-access-kk7nw" (OuterVolumeSpecName: "kube-api-access-kk7nw") pod "ba6fc64a-c1a9-41dd-aa41-4f08230fd589" (UID: "ba6fc64a-c1a9-41dd-aa41-4f08230fd589"). InnerVolumeSpecName "kube-api-access-kk7nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:49.548177 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.548152    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-envoy\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.548177 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.548176    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-token\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.548324 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.548185    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk7nw\" (UniqueName: \"kubernetes.io/projected/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-kube-api-access-kk7nw\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.548324 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.548195    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6fc64a-c1a9-41dd-aa41-4f08230fd589-istio-podinfo\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:28:49.760659 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.760616    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerID="22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c" exitCode=0
Sep 16 12:28:49.760866 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.760682    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"
Sep 16 12:28:49.760866 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.760701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" event={"ID":"ba6fc64a-c1a9-41dd-aa41-4f08230fd589","Type":"ContainerDied","Data":"22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c"}
Sep 16 12:28:49.760866 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.760793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5" event={"ID":"ba6fc64a-c1a9-41dd-aa41-4f08230fd589","Type":"ContainerDied","Data":"af134835fb849ed238fc63449525ad4b58129f0321b42c432955d14df2b1584a"}
Sep 16 12:28:49.760866 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.760825    2578 scope.go:117] "RemoveContainer" containerID="22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c"
Sep 16 12:28:49.771345 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.771328    2578 scope.go:117] "RemoveContainer" containerID="5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf"
Sep 16 12:28:49.780683 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.780660    2578 scope.go:117] "RemoveContainer" containerID="22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c"
Sep 16 12:28:49.780951 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:49.780931    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c\": container with ID starting with 22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c not found: ID does not exist" containerID="22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c"
Sep 16 12:28:49.781052 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.780957    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c"} err="failed to get container status \"22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c\": rpc error: code = NotFound desc = could not find container \"22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c\": container with ID starting with 22ded3ad64195e521e4860f302cd8a681b2e5e79140630c17395671c4753795c not found: ID does not exist"
Sep 16 12:28:49.781052 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.780975    2578 scope.go:117] "RemoveContainer" containerID="5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf"
Sep 16 12:28:49.781272 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:28:49.781258    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf\": container with ID starting with 5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf not found: ID does not exist" containerID="5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf"
Sep 16 12:28:49.781310 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.781276    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf"} err="failed to get container status \"5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf\": rpc error: code = NotFound desc = could not find container \"5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf\": container with ID starting with 5c9053bd7599176e0f5bed768d16e53c7c36563be52c543a1e42936780107bcf not found: ID does not exist"
Sep 16 12:28:49.785787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.785760    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"]
Sep 16 12:28:49.788631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:49.788609    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6f8610-openshift-default-5655bcbdb8-lc4j5"]
Sep 16 12:28:50.886528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:50.886493    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" path="/var/lib/kubelet/pods/ba6fc64a-c1a9-41dd-aa41-4f08230fd589/volumes"
Sep 16 12:28:57.170206 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:57.170172    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:57.170615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:57.170254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:57.182941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:57.182916    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:28:57.814481 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:28:57.814456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:29:13.954921 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.954889    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"]
Sep 16 12:29:13.955567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.955426    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy"
Sep 16 12:29:13.955567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.955446    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy"
Sep 16 12:29:13.955567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.955464    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy"
Sep 16 12:29:13.955567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.955473    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy"
Sep 16 12:29:13.955795 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.955571    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy"
Sep 16 12:29:13.955795 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.955582    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6fc64a-c1a9-41dd-aa41-4f08230fd589" containerName="istio-proxy"
Sep 16 12:29:13.960290 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.960270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:13.963167 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.963140    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"openshift-service-ca.crt\""
Sep 16 12:29:13.964104 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.964080    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"kube-root-ca.crt\""
Sep 16 12:29:13.964331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.964313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"sub-test-model-c03953-kserve-self-signed-certs\""
Sep 16 12:29:13.964657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.964637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"default-dockercfg-m695g\""
Sep 16 12:29:13.966381 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:13.966354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"]
Sep 16 12:29:14.057461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.057435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-dshm\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.057613 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.057464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tmp-dir\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.057613 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.057498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s655j\" (UniqueName: \"kubernetes.io/projected/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-kube-api-access-s655j\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.057613 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.057536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tls-certs\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.057613 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.057595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-model-cache\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.057755 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.057640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-home\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158295 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s655j\" (UniqueName: \"kubernetes.io/projected/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-kube-api-access-s655j\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tls-certs\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-model-cache\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158505 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-home\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158634 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-dshm\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158685 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tmp-dir\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158738 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-model-cache\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158844 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-home\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.158941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.158925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tmp-dir\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.160859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.160840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-dshm\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.161157 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.161139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tls-certs\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.165907 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.165883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s655j\" (UniqueName: \"kubernetes.io/projected/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-kube-api-access-s655j\") pod \"sub-test-model-c03953-kserve-58dff4495c-s2v8h\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") " pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.273679 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.273586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:14.548589 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.548558    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"]
Sep 16 12:29:14.549042 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:29:14.549005    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdcf11c0_4562_4435_bbc8_1cb38f8ad79a.slice/crio-42d7fa30cb656896b8bcc908aee5b847589d3d709d75f6daba3b8891ffb9306d WatchSource:0}: Error finding container 42d7fa30cb656896b8bcc908aee5b847589d3d709d75f6daba3b8891ffb9306d: Status 404 returned error can't find the container with id 42d7fa30cb656896b8bcc908aee5b847589d3d709d75f6daba3b8891ffb9306d
Sep 16 12:29:14.891321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.891218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h" event={"ID":"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a","Type":"ContainerStarted","Data":"50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a"}
Sep 16 12:29:14.891321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.891257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h" event={"ID":"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a","Type":"ContainerStarted","Data":"42d7fa30cb656896b8bcc908aee5b847589d3d709d75f6daba3b8891ffb9306d"}
Sep 16 12:29:14.908252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:14.908172    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h" podStartSLOduration=1.9081549519999998 podStartE2EDuration="1.908154952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:29:14.907248124 +0000 UTC m=+1644.544572962" watchObservedRunningTime="2026-09-16 12:29:14.908154952 +0000 UTC m=+1644.545479791"
Sep 16 12:29:24.273748 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:24.273712    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:24.274235 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:24.273763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:24.286275 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:24.286252    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:24.962714 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:24.962684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:29:48.299232 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.299194    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-eab000"]
Sep 16 12:29:48.302779 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.302761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-eab000"
Sep 16 12:29:48.305451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.305419    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zpzr9\""
Sep 16 12:29:48.307742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.307717    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-eab000"]
Sep 16 12:29:48.349525 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.349492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4w9x\" (UniqueName: \"kubernetes.io/projected/53959186-474a-446d-9f9e-de08ef629e23-kube-api-access-s4w9x\") pod \"test-curl-7478-eab000\" (UID: \"53959186-474a-446d-9f9e-de08ef629e23\") " pod="opendatahub/test-curl-7478-eab000"
Sep 16 12:29:48.450814 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.450779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4w9x\" (UniqueName: \"kubernetes.io/projected/53959186-474a-446d-9f9e-de08ef629e23-kube-api-access-s4w9x\") pod \"test-curl-7478-eab000\" (UID: \"53959186-474a-446d-9f9e-de08ef629e23\") " pod="opendatahub/test-curl-7478-eab000"
Sep 16 12:29:48.458846 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.458818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4w9x\" (UniqueName: \"kubernetes.io/projected/53959186-474a-446d-9f9e-de08ef629e23-kube-api-access-s4w9x\") pod \"test-curl-7478-eab000\" (UID: \"53959186-474a-446d-9f9e-de08ef629e23\") " pod="opendatahub/test-curl-7478-eab000"
Sep 16 12:29:48.614328 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.614252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-eab000"
Sep 16 12:29:48.883926 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:29:48.883898    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53959186_474a_446d_9f9e_de08ef629e23.slice/crio-9eef9e08ab0128f8eb3a0bdcc191843aaeafa9624444ff715df2878099bb0092 WatchSource:0}: Error finding container 9eef9e08ab0128f8eb3a0bdcc191843aaeafa9624444ff715df2878099bb0092: Status 404 returned error can't find the container with id 9eef9e08ab0128f8eb3a0bdcc191843aaeafa9624444ff715df2878099bb0092
Sep 16 12:29:48.885580 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:48.885556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-eab000"]
Sep 16 12:29:49.074484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.074452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-eab000" event={"ID":"53959186-474a-446d-9f9e-de08ef629e23","Type":"ContainerStarted","Data":"c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1"}
Sep 16 12:29:49.074484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.074491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-eab000" event={"ID":"53959186-474a-446d-9f9e-de08ef629e23","Type":"ContainerStarted","Data":"9eef9e08ab0128f8eb3a0bdcc191843aaeafa9624444ff715df2878099bb0092"}
Sep 16 12:29:49.091263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.091211    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7478-eab000" podStartSLOduration=1.091197106 podStartE2EDuration="1.091197106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:29:49.088948459 +0000 UTC m=+1678.726273298" watchObservedRunningTime="2026-09-16 12:29:49.091197106 +0000 UTC m=+1678.728521997"
Sep 16 12:29:49.702034 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.701955    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-eab000"]
Sep 16 12:29:49.705820 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.705791    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-eab000"]
Sep 16 12:29:49.845464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.845432    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-b9d052"]
Sep 16 12:29:49.845790 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.845777    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53959186-474a-446d-9f9e-de08ef629e23" containerName="test-curl-7478-eab000"
Sep 16 12:29:49.845790 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.845791    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53959186-474a-446d-9f9e-de08ef629e23" containerName="test-curl-7478-eab000"
Sep 16 12:29:49.845889 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.845846    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53959186-474a-446d-9f9e-de08ef629e23" containerName="test-curl-7478-eab000"
Sep 16 12:29:49.848739 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.848725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-b9d052"
Sep 16 12:29:49.861917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.861888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lclmd\" (UniqueName: \"kubernetes.io/projected/639256e7-e1fc-43ae-88c2-f6e487099a38-kube-api-access-lclmd\") pod \"test-curl-7478-b9d052\" (UID: \"639256e7-e1fc-43ae-88c2-f6e487099a38\") " pod="opendatahub/test-curl-7478-b9d052"
Sep 16 12:29:49.863077 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.863057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-b9d052"]
Sep 16 12:29:49.962848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.962820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclmd\" (UniqueName: \"kubernetes.io/projected/639256e7-e1fc-43ae-88c2-f6e487099a38-kube-api-access-lclmd\") pod \"test-curl-7478-b9d052\" (UID: \"639256e7-e1fc-43ae-88c2-f6e487099a38\") " pod="opendatahub/test-curl-7478-b9d052"
Sep 16 12:29:49.972142 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:49.972110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclmd\" (UniqueName: \"kubernetes.io/projected/639256e7-e1fc-43ae-88c2-f6e487099a38-kube-api-access-lclmd\") pod \"test-curl-7478-b9d052\" (UID: \"639256e7-e1fc-43ae-88c2-f6e487099a38\") " pod="opendatahub/test-curl-7478-b9d052"
Sep 16 12:29:50.079569 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:29:50.079540    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7478-eab000" container="test-curl-7478-eab000"
Sep 16 12:29:50.157639 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:50.157600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-b9d052"
Sep 16 12:29:50.429181 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:50.429153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-b9d052"]
Sep 16 12:29:50.430084 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:29:50.430047    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod639256e7_e1fc_43ae_88c2_f6e487099a38.slice/crio-cf8648d1cba55d9b073d8a41f502ca4a8c4d04dfbd39442f96e4990a1ed2a96a WatchSource:0}: Error finding container cf8648d1cba55d9b073d8a41f502ca4a8c4d04dfbd39442f96e4990a1ed2a96a: Status 404 returned error can't find the container with id cf8648d1cba55d9b073d8a41f502ca4a8c4d04dfbd39442f96e4990a1ed2a96a
Sep 16 12:29:51.085446 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:51.085397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-b9d052" event={"ID":"639256e7-e1fc-43ae-88c2-f6e487099a38","Type":"ContainerStarted","Data":"40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced"}
Sep 16 12:29:51.085446 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:51.085443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-b9d052" event={"ID":"639256e7-e1fc-43ae-88c2-f6e487099a38","Type":"ContainerStarted","Data":"cf8648d1cba55d9b073d8a41f502ca4a8c4d04dfbd39442f96e4990a1ed2a96a"}
Sep 16 12:29:51.085931 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:51.085467    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7478-eab000" podUID="53959186-474a-446d-9f9e-de08ef629e23" containerName="test-curl-7478-eab000" containerID="cri-o://c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1" gracePeriod=2
Sep 16 12:29:51.099822 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:51.099776    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7478-b9d052" podStartSLOduration=2.099760881 podStartE2EDuration="2.099760881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:29:51.098577796 +0000 UTC m=+1680.735902634" watchObservedRunningTime="2026-09-16 12:29:51.099760881 +0000 UTC m=+1680.737085718"
Sep 16 12:29:51.714767 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:51.714715    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-b9d052"]
Sep 16 12:29:51.717321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:51.717287    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-b9d052"]
Sep 16 12:29:52.090979 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:29:52.090948    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7478-b9d052" container="test-curl-7478-b9d052"
Sep 16 12:29:53.096016 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:53.095983    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7478-b9d052" podUID="639256e7-e1fc-43ae-88c2-f6e487099a38" containerName="test-curl-7478-b9d052" containerID="cri-o://40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced" gracePeriod=2
Sep 16 12:29:53.267366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:53.267341    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-eab000"
Sep 16 12:29:53.292813 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:53.292790    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4w9x\" (UniqueName: \"kubernetes.io/projected/53959186-474a-446d-9f9e-de08ef629e23-kube-api-access-s4w9x\") pod \"53959186-474a-446d-9f9e-de08ef629e23\" (UID: \"53959186-474a-446d-9f9e-de08ef629e23\") "
Sep 16 12:29:53.294849 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:53.294823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53959186-474a-446d-9f9e-de08ef629e23-kube-api-access-s4w9x" (OuterVolumeSpecName: "kube-api-access-s4w9x") pod "53959186-474a-446d-9f9e-de08ef629e23" (UID: "53959186-474a-446d-9f9e-de08ef629e23"). InnerVolumeSpecName "kube-api-access-s4w9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:29:53.394502 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:53.394435    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4w9x\" (UniqueName: \"kubernetes.io/projected/53959186-474a-446d-9f9e-de08ef629e23-kube-api-access-s4w9x\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:29:54.102652 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:54.102619    2578 generic.go:356] "Generic (PLEG): container finished" podID="53959186-474a-446d-9f9e-de08ef629e23" containerID="c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1" exitCode=137
Sep 16 12:29:54.103138 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:54.102660    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-eab000"
Sep 16 12:29:54.103138 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:54.102666    2578 scope.go:117] "RemoveContainer" containerID="c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1"
Sep 16 12:29:54.112225 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:54.112209    2578 scope.go:117] "RemoveContainer" containerID="c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1"
Sep 16 12:29:54.112626 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:29:54.112506    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1\": container with ID starting with c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1 not found: ID does not exist" containerID="c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1"
Sep 16 12:29:54.112670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:54.112637    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1"} err="failed to get container status \"c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1\": rpc error: code = NotFound desc = could not find container \"c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1\": container with ID starting with c86fbce6a8806e5541bbc1e3f2b09fb14fabf3a69e52f46788918d7077b699d1 not found: ID does not exist"
Sep 16 12:29:54.113963 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:29:54.113943    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-eab000\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" podUID="53959186-474a-446d-9f9e-de08ef629e23" pod="opendatahub/test-curl-7478-eab000"
Sep 16 12:29:54.119432 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:29:54.119411    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53959186_474a_446d_9f9e_de08ef629e23.slice/crio-9eef9e08ab0128f8eb3a0bdcc191843aaeafa9624444ff715df2878099bb0092\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53959186_474a_446d_9f9e_de08ef629e23.slice\": RecentStats: unable to find data in memory cache]"
Sep 16 12:29:54.884304 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:54.884272    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53959186-474a-446d-9f9e-de08ef629e23" path="/var/lib/kubelet/pods/53959186-474a-446d-9f9e-de08ef629e23/volumes"
Sep 16 12:29:55.278355 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:55.278329    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-b9d052"
Sep 16 12:29:55.308461 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:55.308431    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lclmd\" (UniqueName: \"kubernetes.io/projected/639256e7-e1fc-43ae-88c2-f6e487099a38-kube-api-access-lclmd\") pod \"639256e7-e1fc-43ae-88c2-f6e487099a38\" (UID: \"639256e7-e1fc-43ae-88c2-f6e487099a38\") "
Sep 16 12:29:55.310717 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:55.310681    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639256e7-e1fc-43ae-88c2-f6e487099a38-kube-api-access-lclmd" (OuterVolumeSpecName: "kube-api-access-lclmd") pod "639256e7-e1fc-43ae-88c2-f6e487099a38" (UID: "639256e7-e1fc-43ae-88c2-f6e487099a38"). InnerVolumeSpecName "kube-api-access-lclmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:29:55.409998 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:55.409903    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lclmd\" (UniqueName: \"kubernetes.io/projected/639256e7-e1fc-43ae-88c2-f6e487099a38-kube-api-access-lclmd\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:29:56.115615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:56.115566    2578 generic.go:356] "Generic (PLEG): container finished" podID="639256e7-e1fc-43ae-88c2-f6e487099a38" containerID="40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced" exitCode=137
Sep 16 12:29:56.115615 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:56.115601    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-b9d052"
Sep 16 12:29:56.115871 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:56.115648    2578 scope.go:117] "RemoveContainer" containerID="40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced"
Sep 16 12:29:56.127751 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:56.127736    2578 scope.go:117] "RemoveContainer" containerID="40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced"
Sep 16 12:29:56.128054 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:29:56.128029    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced\": container with ID starting with 40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced not found: ID does not exist" containerID="40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced"
Sep 16 12:29:56.128109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:56.128067    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced"} err="failed to get container status \"40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced\": rpc error: code = NotFound desc = could not find container \"40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced\": container with ID starting with 40f36d6b4703292192dffe2d48d945063abed1b8bce4b78bdeb9ad5fac9a6ced not found: ID does not exist"
Sep 16 12:29:56.129330 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:29:56.129304    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-b9d052\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" podUID="639256e7-e1fc-43ae-88c2-f6e487099a38" pod="opendatahub/test-curl-7478-b9d052"
Sep 16 12:29:56.884150 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:56.884119    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639256e7-e1fc-43ae-88c2-f6e487099a38" path="/var/lib/kubelet/pods/639256e7-e1fc-43ae-88c2-f6e487099a38/volumes"
Sep 16 12:29:58.940880 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:58.940847    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"]
Sep 16 12:29:59.962596 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:29:59.962563    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"]
Sep 16 12:30:00.133091 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.133057    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"]
Sep 16 12:30:00.133407 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.133395    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639256e7-e1fc-43ae-88c2-f6e487099a38" containerName="test-curl-7478-b9d052"
Sep 16 12:30:00.133456 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.133408    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="639256e7-e1fc-43ae-88c2-f6e487099a38" containerName="test-curl-7478-b9d052"
Sep 16 12:30:00.133557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.133535    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="639256e7-e1fc-43ae-88c2-f6e487099a38" containerName="test-curl-7478-b9d052"
Sep 16 12:30:00.137839 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.137820    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"
Sep 16 12:30:00.140592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.140569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lc86\""
Sep 16 12:30:00.147228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.147209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"]
Sep 16 12:30:00.149312 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.149294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"
Sep 16 12:30:00.424592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:00.424562    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"]
Sep 16 12:30:00.426155 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:30:00.426132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258f9816_044c_418b_8575_7d6893f8269f.slice/crio-05b0f5bbd9287896999284fe50a0a8b689cced6d926b5320524dd62b4789688c WatchSource:0}: Error finding container 05b0f5bbd9287896999284fe50a0a8b689cced6d926b5320524dd62b4789688c: Status 404 returned error can't find the container with id 05b0f5bbd9287896999284fe50a0a8b689cced6d926b5320524dd62b4789688c
Sep 16 12:30:01.146117 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:01.146083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh" event={"ID":"258f9816-044c-418b-8575-7d6893f8269f","Type":"ContainerStarted","Data":"05b0f5bbd9287896999284fe50a0a8b689cced6d926b5320524dd62b4789688c"}
Sep 16 12:30:04.164701 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:04.164665    2578 generic.go:356] "Generic (PLEG): container finished" podID="258f9816-044c-418b-8575-7d6893f8269f" containerID="5c5d5063f5a82479a9255cb136fdba5660e015a36f2f93e3f232b57c7e5f9449" exitCode=0
Sep 16 12:30:04.165104 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:04.164757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh" event={"ID":"258f9816-044c-418b-8575-7d6893f8269f","Type":"ContainerDied","Data":"5c5d5063f5a82479a9255cb136fdba5660e015a36f2f93e3f232b57c7e5f9449"}
Sep 16 12:30:05.296395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:05.296370    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"
Sep 16 12:30:06.177579 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:06.177542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh" event={"ID":"258f9816-044c-418b-8575-7d6893f8269f","Type":"ContainerDied","Data":"05b0f5bbd9287896999284fe50a0a8b689cced6d926b5320524dd62b4789688c"}
Sep 16 12:30:06.177579 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:06.177576    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"
Sep 16 12:30:06.177795 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:06.177614    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05b0f5bbd9287896999284fe50a0a8b689cced6d926b5320524dd62b4789688c"
Sep 16 12:30:13.960773 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:13.960667    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt" podUID="5b04e141-d672-4198-af0a-7f83ad01b14a" containerName="main" containerID="cri-o://358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e" gracePeriod=45
Sep 16 12:30:14.199610 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.199586    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:30:14.222827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.222746    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b04e141-d672-4198-af0a-7f83ad01b14a" containerID="358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e" exitCode=0
Sep 16 12:30:14.222827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.222791    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"
Sep 16 12:30:14.223012 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.222829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt" event={"ID":"5b04e141-d672-4198-af0a-7f83ad01b14a","Type":"ContainerDied","Data":"358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e"}
Sep 16 12:30:14.223012 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.222893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt" event={"ID":"5b04e141-d672-4198-af0a-7f83ad01b14a","Type":"ContainerDied","Data":"2486db84cef229275aefcced9bd015b5fdc3e13b000354f115a1fdfab765c364"}
Sep 16 12:30:14.223012 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.222916    2578 scope.go:117] "RemoveContainer" containerID="358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e"
Sep 16 12:30:14.233273 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.233255    2578 scope.go:117] "RemoveContainer" containerID="358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e"
Sep 16 12:30:14.233575 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:30:14.233556    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e\": container with ID starting with 358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e not found: ID does not exist" containerID="358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e"
Sep 16 12:30:14.233651 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.233587    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e"} err="failed to get container status \"358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e\": rpc error: code = NotFound desc = could not find container \"358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e\": container with ID starting with 358c7a8c856ad734eb4c74fd154a12d963cd797d6faf6b77250c9145e724591e not found: ID does not exist"
Sep 16 12:30:14.277034 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277010    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-tmp-dir\") pod \"5b04e141-d672-4198-af0a-7f83ad01b14a\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") "
Sep 16 12:30:14.277138 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-dshm\") pod \"5b04e141-d672-4198-af0a-7f83ad01b14a\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") "
Sep 16 12:30:14.277138 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-home\") pod \"5b04e141-d672-4198-af0a-7f83ad01b14a\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") "
Sep 16 12:30:14.277138 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fndf8\" (UniqueName: \"kubernetes.io/projected/5b04e141-d672-4198-af0a-7f83ad01b14a-kube-api-access-fndf8\") pod \"5b04e141-d672-4198-af0a-7f83ad01b14a\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") "
Sep 16 12:30:14.277241 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277160    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-model-cache\") pod \"5b04e141-d672-4198-af0a-7f83ad01b14a\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") "
Sep 16 12:30:14.277241 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277223    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04e141-d672-4198-af0a-7f83ad01b14a-tls-certs\") pod \"5b04e141-d672-4198-af0a-7f83ad01b14a\" (UID: \"5b04e141-d672-4198-af0a-7f83ad01b14a\") "
Sep 16 12:30:14.277318 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277282    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b04e141-d672-4198-af0a-7f83ad01b14a" (UID: "5b04e141-d672-4198-af0a-7f83ad01b14a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:14.277408 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-home" (OuterVolumeSpecName: "home") pod "5b04e141-d672-4198-af0a-7f83ad01b14a" (UID: "5b04e141-d672-4198-af0a-7f83ad01b14a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:14.277476 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b04e141-d672-4198-af0a-7f83ad01b14a" (UID: "5b04e141-d672-4198-af0a-7f83ad01b14a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:14.277618 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277596    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:14.277618 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277618    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:14.277759 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.277629    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:14.279464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.279437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-dshm" (OuterVolumeSpecName: "dshm") pod "5b04e141-d672-4198-af0a-7f83ad01b14a" (UID: "5b04e141-d672-4198-af0a-7f83ad01b14a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:14.279573 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.279462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b04e141-d672-4198-af0a-7f83ad01b14a-kube-api-access-fndf8" (OuterVolumeSpecName: "kube-api-access-fndf8") pod "5b04e141-d672-4198-af0a-7f83ad01b14a" (UID: "5b04e141-d672-4198-af0a-7f83ad01b14a"). InnerVolumeSpecName "kube-api-access-fndf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:30:14.279573 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.279559    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b04e141-d672-4198-af0a-7f83ad01b14a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b04e141-d672-4198-af0a-7f83ad01b14a" (UID: "5b04e141-d672-4198-af0a-7f83ad01b14a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:30:14.378120 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.378076    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04e141-d672-4198-af0a-7f83ad01b14a-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:14.378120 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.378114    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04e141-d672-4198-af0a-7f83ad01b14a-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:14.378120 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.378123    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fndf8\" (UniqueName: \"kubernetes.io/projected/5b04e141-d672-4198-af0a-7f83ad01b14a-kube-api-access-fndf8\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:14.544242 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.544208    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"]
Sep 16 12:30:14.547712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.547687    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/sub-test-model-db42a6-kserve-59947cbb65-4hhbt"]
Sep 16 12:30:14.884290 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.884193    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b04e141-d672-4198-af0a-7f83ad01b14a" path="/var/lib/kubelet/pods/5b04e141-d672-4198-af0a-7f83ad01b14a/volumes"
Sep 16 12:30:14.982453 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:14.982420    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h" podUID="bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" containerName="main" containerID="cri-o://50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a" gracePeriod=45
Sep 16 12:30:15.226557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.226533    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:30:15.229771 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.229744    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"
Sep 16 12:30:15.229911 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.229783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h" event={"ID":"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a","Type":"ContainerDied","Data":"50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a"}
Sep 16 12:30:15.229911 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.229833    2578 scope.go:117] "RemoveContainer" containerID="50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a"
Sep 16 12:30:15.229911 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.229781    2578 generic.go:356] "Generic (PLEG): container finished" podID="bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" containerID="50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a" exitCode=0
Sep 16 12:30:15.230055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.229985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h" event={"ID":"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a","Type":"ContainerDied","Data":"42d7fa30cb656896b8bcc908aee5b847589d3d709d75f6daba3b8891ffb9306d"}
Sep 16 12:30:15.238402 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.238382    2578 scope.go:117] "RemoveContainer" containerID="50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a"
Sep 16 12:30:15.238695 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:30:15.238673    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a\": container with ID starting with 50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a not found: ID does not exist" containerID="50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a"
Sep 16 12:30:15.238787 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.238702    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a"} err="failed to get container status \"50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a\": rpc error: code = NotFound desc = could not find container \"50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a\": container with ID starting with 50b2694b9f5f6adfe7eabf123ce8d7a71a2f69f76d6bc861643aa61246ad819a not found: ID does not exist"
Sep 16 12:30:15.284524 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.284495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-model-cache\") pod \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") "
Sep 16 12:30:15.284688 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.284539    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-home\") pod \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") "
Sep 16 12:30:15.284688 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.284584    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s655j\" (UniqueName: \"kubernetes.io/projected/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-kube-api-access-s655j\") pod \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") "
Sep 16 12:30:15.284688 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.284618    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tls-certs\") pod \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") "
Sep 16 12:30:15.284688 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.284673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-dshm\") pod \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") "
Sep 16 12:30:15.284898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.284722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tmp-dir\") pod \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\" (UID: \"bdcf11c0-4562-4435-bbc8-1cb38f8ad79a\") "
Sep 16 12:30:15.284898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.284804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" (UID: "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:15.284898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.284855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-home" (OuterVolumeSpecName: "home") pod "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" (UID: "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:15.285078 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.285026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" (UID: "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:15.285279 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.285256    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:15.285401 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.285283    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:15.285401 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.285296    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:15.287087 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.287066    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" (UID: "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:30:15.287087 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.287080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-dshm" (OuterVolumeSpecName: "dshm") pod "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" (UID: "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:15.287234 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.287161    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-kube-api-access-s655j" (OuterVolumeSpecName: "kube-api-access-s655j") pod "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" (UID: "bdcf11c0-4562-4435-bbc8-1cb38f8ad79a"). InnerVolumeSpecName "kube-api-access-s655j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:30:15.386536 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.386480    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s655j\" (UniqueName: \"kubernetes.io/projected/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-kube-api-access-s655j\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:15.386536 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.386531    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:15.386536 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.386541    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:30:15.550821 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.550789    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"]
Sep 16 12:30:15.553054 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:15.553028    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/sub-test-model-c03953-kserve-58dff4495c-s2v8h"]
Sep 16 12:30:16.890058 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:16.890024    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" path="/var/lib/kubelet/pods/bdcf11c0-4562-4435-bbc8-1cb38f8ad79a/volumes"
Sep 16 12:30:17.755044 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755007    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"]
Sep 16 12:30:17.755492 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755478    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" containerName="main"
Sep 16 12:30:17.755552 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755497    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" containerName="main"
Sep 16 12:30:17.755552 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755533    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258f9816-044c-418b-8575-7d6893f8269f" containerName="cleanup"
Sep 16 12:30:17.755552 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755543    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="258f9816-044c-418b-8575-7d6893f8269f" containerName="cleanup"
Sep 16 12:30:17.755652 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755582    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b04e141-d672-4198-af0a-7f83ad01b14a" containerName="main"
Sep 16 12:30:17.755652 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755591    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b04e141-d672-4198-af0a-7f83ad01b14a" containerName="main"
Sep 16 12:30:17.755716 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755669    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="258f9816-044c-418b-8575-7d6893f8269f" containerName="cleanup"
Sep 16 12:30:17.755716 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755682    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b04e141-d672-4198-af0a-7f83ad01b14a" containerName="main"
Sep 16 12:30:17.755716 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.755693    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdcf11c0-4562-4435-bbc8-1cb38f8ad79a" containerName="main"
Sep 16 12:30:17.760099 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.760079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:17.764083 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.764061    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-db42a6\"/\"kube-root-ca.crt\""
Sep 16 12:30:17.764225 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.764061    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db42a6\"/\"rate-test-model-db42a6-kserve-self-signed-certs\""
Sep 16 12:30:17.764225 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.764068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-db42a6\"/\"default-dockercfg-czqpt\""
Sep 16 12:30:17.764366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.764079    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-db42a6\"/\"openshift-service-ca.crt\""
Sep 16 12:30:17.767157 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.767137    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"]
Sep 16 12:30:17.910220 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.910189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tls-certs\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:17.910673 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.910231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtwr\" (UniqueName: \"kubernetes.io/projected/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-kube-api-access-9qtwr\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:17.910673 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.910260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-dshm\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:17.910673 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.910328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tmp-dir\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:17.910673 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.910392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-home\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:17.910673 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:17.910429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-model-cache\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.011549 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.011429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-home\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.011549 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.011483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-model-cache\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.011549 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.011536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tls-certs\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.011848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.011565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtwr\" (UniqueName: \"kubernetes.io/projected/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-kube-api-access-9qtwr\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.011848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.011617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-dshm\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.011848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.011641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tmp-dir\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.011991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.011916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-home\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.011991 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.011961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tmp-dir\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.012119 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.012004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-model-cache\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.014016 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.013992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-dshm\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.014133 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.014109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tls-certs\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.020260 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.020234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtwr\" (UniqueName: \"kubernetes.io/projected/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-kube-api-access-9qtwr\") pod \"rate-test-model-db42a6-kserve-57c76c78f5-s2nx6\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") " pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.070657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.070634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:18.344555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:18.344498    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"]
Sep 16 12:30:18.345185 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:30:18.345160    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1ae1c30_2bbf_4579_9326_f1b1e93ba5aa.slice/crio-9888957a9dc9e2d19ec1c75df249a012e91220c4d78889371e5765301ce20e1e WatchSource:0}: Error finding container 9888957a9dc9e2d19ec1c75df249a012e91220c4d78889371e5765301ce20e1e: Status 404 returned error can't find the container with id 9888957a9dc9e2d19ec1c75df249a012e91220c4d78889371e5765301ce20e1e
Sep 16 12:30:19.252900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:19.252865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6" event={"ID":"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa","Type":"ContainerStarted","Data":"a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413"}
Sep 16 12:30:19.253374 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:19.252911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6" event={"ID":"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa","Type":"ContainerStarted","Data":"9888957a9dc9e2d19ec1c75df249a012e91220c4d78889371e5765301ce20e1e"}
Sep 16 12:30:19.272550 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:19.272487    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6" podStartSLOduration=2.2724724 podStartE2EDuration="2.2724724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:19.270666548 +0000 UTC m=+1708.907991404" watchObservedRunningTime="2026-09-16 12:30:19.2724724 +0000 UTC m=+1708.909797238"
Sep 16 12:30:28.071301 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:28.071256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:28.071835 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:28.071319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:28.083790 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:28.083769    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:30:28.314464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:30:28.314436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:31:31.334123 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.334088    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"]
Sep 16 12:31:31.336331 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.336315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.339780 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:31.339745    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-m695g\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-w2-c03953\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"default-dockercfg-m695g\"" type="*v1.Secret"
Sep 16 12:31:31.339989 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:31.339968    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-w2-c03953\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" podUID="c35967bf-d19e-4b7c-9e08-9a6852815696" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.340245 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:31.340222    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-w2-c03953\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 12:31:31.340379 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:31.340363    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-w2-c03953\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 12:31:31.340450 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:31.340386    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"rate-test-model-c03953-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-136-170.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-w2-c03953\": no relationship found between node 'ip-10-0-136-170.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"rate-test-model-c03953-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 16 12:31:31.354081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.354056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.354319 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.354115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-dshm\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.354319 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.354143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpxv\" (UniqueName: \"kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.354319 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.354167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-home\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.354319 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.354219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-model-cache\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.354319 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.354263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-tmp-dir\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.358032 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.357994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"]
Sep 16 12:31:31.454983 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.454948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.455188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.455004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-dshm\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.455188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.455036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpxv\" (UniqueName: \"kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.455188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.455052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-home\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.455188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.455082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-model-cache\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.455399 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.455218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-tmp-dir\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.455470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.455449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-model-cache\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.455559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.455461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-home\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.455619 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.455575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-tmp-dir\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:31.457601 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:31.457573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-dshm\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:32.167015 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:32.166979    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"openshift-service-ca.crt\""
Sep 16 12:31:32.267354 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:32.267323    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"default-dockercfg-m695g\""
Sep 16 12:31:32.455851 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:32.455830    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Sep 16 12:31:32.456210 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:32.455909    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs podName:c35967bf-d19e-4b7c-9e08-9a6852815696 nodeName:}" failed. No retries permitted until 2026-09-16 12:31:32.955892835 +0000 UTC m=+1782.593217652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs") pod "rate-test-model-c03953-kserve-56c67c846c-sjpc8" (UID: "c35967bf-d19e-4b7c-9e08-9a6852815696") : failed to sync secret cache: timed out waiting for the condition
Sep 16 12:31:32.460774 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:32.460760    2578 projected.go:291] Couldn't get configMap ai-tenant-e2e-shared-b-w2-c03953/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 16 12:31:32.460828 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:32.460789    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4mpxv for pod ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8: failed to sync configmap cache: timed out waiting for the condition
Sep 16 12:31:32.460868 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:31:32.460829    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv podName:c35967bf-d19e-4b7c-9e08-9a6852815696 nodeName:}" failed. No retries permitted until 2026-09-16 12:31:32.960816899 +0000 UTC m=+1782.598141715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4mpxv" (UniqueName: "kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv") pod "rate-test-model-c03953-kserve-56c67c846c-sjpc8" (UID: "c35967bf-d19e-4b7c-9e08-9a6852815696") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 12:31:32.654761 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:32.654730    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"kube-root-ca.crt\""
Sep 16 12:31:32.896908 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:32.896820    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"rate-test-model-c03953-kserve-self-signed-certs\""
Sep 16 12:31:32.967937 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:32.967901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:32.968105 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:32.967970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpxv\" (UniqueName: \"kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:32.970417 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:32.970391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:32.970592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:32.970575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpxv\" (UniqueName: \"kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv\") pod \"rate-test-model-c03953-kserve-56c67c846c-sjpc8\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") " pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:33.145601 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:33.145567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:33.413298 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:33.413204    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"]
Sep 16 12:31:33.416144 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:31:33.416106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc35967bf_d19e_4b7c_9e08_9a6852815696.slice/crio-d717514e918cf150697386d2452fb25521571b93c3e52239efb33bdc70e92713 WatchSource:0}: Error finding container d717514e918cf150697386d2452fb25521571b93c3e52239efb33bdc70e92713: Status 404 returned error can't find the container with id d717514e918cf150697386d2452fb25521571b93c3e52239efb33bdc70e92713
Sep 16 12:31:33.647970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:33.647934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8" event={"ID":"c35967bf-d19e-4b7c-9e08-9a6852815696","Type":"ContainerStarted","Data":"c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb"}
Sep 16 12:31:33.648436 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:33.647979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8" event={"ID":"c35967bf-d19e-4b7c-9e08-9a6852815696","Type":"ContainerStarted","Data":"d717514e918cf150697386d2452fb25521571b93c3e52239efb33bdc70e92713"}
Sep 16 12:31:33.664934 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:33.664838    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8" podStartSLOduration=2.664822423 podStartE2EDuration="2.664822423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:33.662654613 +0000 UTC m=+1783.299979642" watchObservedRunningTime="2026-09-16 12:31:33.664822423 +0000 UTC m=+1783.302147261"
Sep 16 12:31:43.146161 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:43.146069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:43.146161 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:43.146122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:43.158940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:43.158910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:43.711732 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:43.711702    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:31:51.001037 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:51.001012    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:31:51.008579 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:31:51.008554    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:32:21.246508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:21.246475    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"]
Sep 16 12:32:22.148137 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:22.148097    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"]
Sep 16 12:32:28.185025 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.184984    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"]
Sep 16 12:32:28.187636 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.187613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.190445 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.190422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a4f954-serving-cert\""
Sep 16 12:32:28.202772 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.202748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"]
Sep 16 12:32:28.243803 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.243774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqjzz\" (UniqueName: \"kubernetes.io/projected/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-kube-api-access-vqjzz\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.243945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.243827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.243945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.243850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.244062 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.243929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.345227 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.345196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.345399 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.345239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.345399 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.345271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.345399 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.345341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjzz\" (UniqueName: \"kubernetes.io/projected/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-kube-api-access-vqjzz\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.345399 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:32:28.345354    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-serving-cert: secret "maas-api-e2e-ipp-a-a4f954-serving-cert" not found
Sep 16 12:32:28.345584 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:32:28.345438    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls podName:d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff nodeName:}" failed. No retries permitted until 2026-09-16 12:32:28.845422515 +0000 UTC m=+1838.482747331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls") pod "maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb" (UID: "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff") : secret "maas-api-e2e-ipp-a-a4f954-serving-cert" not found
Sep 16 12:32:28.346007 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.345979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.347824 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.347807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-metrics-certs\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.356305 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.356285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjzz\" (UniqueName: \"kubernetes.io/projected/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-kube-api-access-vqjzz\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.850450 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.850415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:28.852865 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:28.852846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:29.098497 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:29.098460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:29.367628 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:29.367596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"]
Sep 16 12:32:29.368965 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:32:29.368944    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f8f74b_8706_4bd2_8b0c_7a56fdc99dff.slice/crio-fff2abf4122a257051aa7e71fe23f5f8c9a1b408684d397aa19d072fca4712de WatchSource:0}: Error finding container fff2abf4122a257051aa7e71fe23f5f8c9a1b408684d397aa19d072fca4712de: Status 404 returned error can't find the container with id fff2abf4122a257051aa7e71fe23f5f8c9a1b408684d397aa19d072fca4712de
Sep 16 12:32:29.370350 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:29.370334    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:32:29.953230 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:29.953195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb" event={"ID":"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff","Type":"ContainerStarted","Data":"2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423"}
Sep 16 12:32:29.953422 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:29.953255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb" event={"ID":"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff","Type":"ContainerStarted","Data":"fff2abf4122a257051aa7e71fe23f5f8c9a1b408684d397aa19d072fca4712de"}
Sep 16 12:32:29.953422 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:29.953281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:29.979548 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:29.979482    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb" podStartSLOduration=1.6132023599999998 podStartE2EDuration="1.979466197s" totalImagesPullingTime="366.263837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:32:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:32:29.977150766 +0000 UTC m=+1839.614475601" watchObservedRunningTime="2026-09-16 12:32:29.979466197 +0000 UTC m=+1839.616791034"
Sep 16 12:32:35.964144 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:35.964112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:32:36.265503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.265392    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6" podUID="f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" containerName="main" containerID="cri-o://a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413" gracePeriod=45
Sep 16 12:32:36.510609 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.510587    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:32:36.617760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.617664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-model-cache\") pod \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") "
Sep 16 12:32:36.617760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.617735    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tmp-dir\") pod \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") "
Sep 16 12:32:36.618002 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.617781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qtwr\" (UniqueName: \"kubernetes.io/projected/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-kube-api-access-9qtwr\") pod \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") "
Sep 16 12:32:36.618002 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.617810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-dshm\") pod \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") "
Sep 16 12:32:36.618002 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.617824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-home\") pod \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") "
Sep 16 12:32:36.618002 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.617865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tls-certs\") pod \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\" (UID: \"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa\") "
Sep 16 12:32:36.618216 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.617998    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" (UID: "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:36.618216 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.618016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" (UID: "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:36.618216 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.618167    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-home" (OuterVolumeSpecName: "home") pod "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" (UID: "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:36.618216 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.618176    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:36.618216 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.618197    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:36.620183 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.620157    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-dshm" (OuterVolumeSpecName: "dshm") pod "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" (UID: "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:36.620298 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.620197    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" (UID: "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:36.620298 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.620246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-kube-api-access-9qtwr" (OuterVolumeSpecName: "kube-api-access-9qtwr") pod "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" (UID: "f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa"). InnerVolumeSpecName "kube-api-access-9qtwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:36.718842 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.718804    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:36.718842 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.718837    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtwr\" (UniqueName: \"kubernetes.io/projected/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-kube-api-access-9qtwr\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:36.718842 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.718850    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:36.719097 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.718862    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:36.991208 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.991176    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" containerID="a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413" exitCode=0
Sep 16 12:32:36.991721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.991222    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"
Sep 16 12:32:36.991721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.991262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6" event={"ID":"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa","Type":"ContainerDied","Data":"a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413"}
Sep 16 12:32:36.991721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.991314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6" event={"ID":"f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa","Type":"ContainerDied","Data":"9888957a9dc9e2d19ec1c75df249a012e91220c4d78889371e5765301ce20e1e"}
Sep 16 12:32:36.991721 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:36.991340    2578 scope.go:117] "RemoveContainer" containerID="a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413"
Sep 16 12:32:37.000760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.000742    2578 scope.go:117] "RemoveContainer" containerID="a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413"
Sep 16 12:32:37.001018 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:32:37.000992    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413\": container with ID starting with a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413 not found: ID does not exist" containerID="a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413"
Sep 16 12:32:37.001135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.001028    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413"} err="failed to get container status \"a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413\": rpc error: code = NotFound desc = could not find container \"a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413\": container with ID starting with a977cd89fbaa857b563e0fa2486c5175f7fe4e001dd79bc8e3a9c8a656bfd413 not found: ID does not exist"
Sep 16 12:32:37.008769 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.008747    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"]
Sep 16 12:32:37.013390 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.013369    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-db42a6/rate-test-model-db42a6-kserve-57c76c78f5-s2nx6"]
Sep 16 12:32:37.167485 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.167403    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8" podUID="c35967bf-d19e-4b7c-9e08-9a6852815696" containerName="main" containerID="cri-o://c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb" gracePeriod=45
Sep 16 12:32:37.411898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.411864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:32:37.526329 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-dshm\") pod \"c35967bf-d19e-4b7c-9e08-9a6852815696\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") "
Sep 16 12:32:37.526508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mpxv\" (UniqueName: \"kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv\") pod \"c35967bf-d19e-4b7c-9e08-9a6852815696\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") "
Sep 16 12:32:37.526508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526395    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-model-cache\") pod \"c35967bf-d19e-4b7c-9e08-9a6852815696\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") "
Sep 16 12:32:37.526508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526426    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-home\") pod \"c35967bf-d19e-4b7c-9e08-9a6852815696\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") "
Sep 16 12:32:37.526508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs\") pod \"c35967bf-d19e-4b7c-9e08-9a6852815696\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") "
Sep 16 12:32:37.526508 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-tmp-dir\") pod \"c35967bf-d19e-4b7c-9e08-9a6852815696\" (UID: \"c35967bf-d19e-4b7c-9e08-9a6852815696\") "
Sep 16 12:32:37.526801 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526709    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-model-cache" (OuterVolumeSpecName: "model-cache") pod "c35967bf-d19e-4b7c-9e08-9a6852815696" (UID: "c35967bf-d19e-4b7c-9e08-9a6852815696"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:37.526801 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-home" (OuterVolumeSpecName: "home") pod "c35967bf-d19e-4b7c-9e08-9a6852815696" (UID: "c35967bf-d19e-4b7c-9e08-9a6852815696"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:37.526879 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.526825    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c35967bf-d19e-4b7c-9e08-9a6852815696" (UID: "c35967bf-d19e-4b7c-9e08-9a6852815696"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:37.528723 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.528702    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-dshm" (OuterVolumeSpecName: "dshm") pod "c35967bf-d19e-4b7c-9e08-9a6852815696" (UID: "c35967bf-d19e-4b7c-9e08-9a6852815696"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:37.528808 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.528720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv" (OuterVolumeSpecName: "kube-api-access-4mpxv") pod "c35967bf-d19e-4b7c-9e08-9a6852815696" (UID: "c35967bf-d19e-4b7c-9e08-9a6852815696"). InnerVolumeSpecName "kube-api-access-4mpxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:37.528849 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.528813    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c35967bf-d19e-4b7c-9e08-9a6852815696" (UID: "c35967bf-d19e-4b7c-9e08-9a6852815696"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:37.628017 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.627989    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpxv\" (UniqueName: \"kubernetes.io/projected/c35967bf-d19e-4b7c-9e08-9a6852815696-kube-api-access-4mpxv\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:37.628017 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.628014    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:37.628168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.628028    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:37.628168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.628038    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35967bf-d19e-4b7c-9e08-9a6852815696-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:37.628168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.628048    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:37.628168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.628058    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35967bf-d19e-4b7c-9e08-9a6852815696-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:32:37.997571 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.997538    2578 generic.go:356] "Generic (PLEG): container finished" podID="c35967bf-d19e-4b7c-9e08-9a6852815696" containerID="c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb" exitCode=0
Sep 16 12:32:37.998045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.997583    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"
Sep 16 12:32:37.998045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.997620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8" event={"ID":"c35967bf-d19e-4b7c-9e08-9a6852815696","Type":"ContainerDied","Data":"c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb"}
Sep 16 12:32:37.998045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.997658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8" event={"ID":"c35967bf-d19e-4b7c-9e08-9a6852815696","Type":"ContainerDied","Data":"d717514e918cf150697386d2452fb25521571b93c3e52239efb33bdc70e92713"}
Sep 16 12:32:37.998045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:37.997676    2578 scope.go:117] "RemoveContainer" containerID="c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb"
Sep 16 12:32:38.008181 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:38.008165    2578 scope.go:117] "RemoveContainer" containerID="c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb"
Sep 16 12:32:38.008489 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:32:38.008470    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb\": container with ID starting with c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb not found: ID does not exist" containerID="c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb"
Sep 16 12:32:38.008567 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:38.008499    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb"} err="failed to get container status \"c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb\": rpc error: code = NotFound desc = could not find container \"c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb\": container with ID starting with c79ed6c375d657a10eb376a0c77aa3a0defb24a645fb8769189d6e8224e7d7cb not found: ID does not exist"
Sep 16 12:32:38.020644 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:38.020622    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"]
Sep 16 12:32:38.024859 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:38.024837    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/rate-test-model-c03953-kserve-56c67c846c-sjpc8"]
Sep 16 12:32:38.884377 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:38.884344    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c35967bf-d19e-4b7c-9e08-9a6852815696" path="/var/lib/kubelet/pods/c35967bf-d19e-4b7c-9e08-9a6852815696/volumes"
Sep 16 12:32:38.884778 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:38.884763    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" path="/var/lib/kubelet/pods/f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa/volumes"
Sep 16 12:32:52.127116 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.127070    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"]
Sep 16 12:32:52.127507 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.127430    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" containerName="main"
Sep 16 12:32:52.127507 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.127442    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" containerName="main"
Sep 16 12:32:52.127507 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.127460    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35967bf-d19e-4b7c-9e08-9a6852815696" containerName="main"
Sep 16 12:32:52.127507 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.127466    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35967bf-d19e-4b7c-9e08-9a6852815696" containerName="main"
Sep 16 12:32:52.127679 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.127560    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1ae1c30-2bbf-4579-9326-f1b1e93ba5aa" containerName="main"
Sep 16 12:32:52.127679 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.127568    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35967bf-d19e-4b7c-9e08-9a6852815696" containerName="main"
Sep 16 12:32:52.129810 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.129791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.132359 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.132336    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3863e0-openshift-default-dockercfg-85mgm\""
Sep 16 12:32:52.141612 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.141590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"]
Sep 16 12:32:52.259835 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.259803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-credential-socket\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.259835 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.259836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-envoy\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.260055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.259871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-podinfo\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.260055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.259896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-socket\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.260055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.259914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776542f7-9926-46eb-94ca-21b473d3f7bc-istiod-ca-cert\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.260055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.259933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-data\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.260055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.259965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-token\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.260055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.260004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffrkj\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-kube-api-access-ffrkj\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.260055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.260040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-certs\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.360729 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.360696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-certs\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.360900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.360740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-credential-socket\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.360900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.360761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-envoy\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.360900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.360804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-podinfo\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.360900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.360830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-socket\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.360964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776542f7-9926-46eb-94ca-21b473d3f7bc-istiod-ca-cert\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.361022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-data\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.361069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-token\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.361093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-certs\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361361 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.361152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffrkj\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-kube-api-access-ffrkj\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361361 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.361167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-credential-socket\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361361 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.361302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-socket\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361507 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.361415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-data\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.361766 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.361716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776542f7-9926-46eb-94ca-21b473d3f7bc-istiod-ca-cert\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.363366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.363342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-envoy\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.363584 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.363566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-podinfo\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.369071 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.369037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-token\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.369269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.369252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffrkj\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-kube-api-access-ffrkj\") pod \"e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") " pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.441034 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.440968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:52.719023 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.718994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"]
Sep 16 12:32:52.720497 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:32:52.720471    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776542f7_9926_46eb_94ca_21b473d3f7bc.slice/crio-ca039cc95e367d965fdecb7babc437b6dc5987bc69614f58eef8180dfc85a02f WatchSource:0}: Error finding container ca039cc95e367d965fdecb7babc437b6dc5987bc69614f58eef8180dfc85a02f: Status 404 returned error can't find the container with id ca039cc95e367d965fdecb7babc437b6dc5987bc69614f58eef8180dfc85a02f
Sep 16 12:32:52.722990 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.722960    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:32:52.723073 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.723019    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:32:52.723073 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:52.723045    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:32:53.082151 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:53.082068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" event={"ID":"776542f7-9926-46eb-94ca-21b473d3f7bc","Type":"ContainerStarted","Data":"8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262"}
Sep 16 12:32:53.082151 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:53.082120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" event={"ID":"776542f7-9926-46eb-94ca-21b473d3f7bc","Type":"ContainerStarted","Data":"ca039cc95e367d965fdecb7babc437b6dc5987bc69614f58eef8180dfc85a02f"}
Sep 16 12:32:53.103124 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:53.103078    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podStartSLOduration=1.10306321 podStartE2EDuration="1.10306321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:32:53.101319418 +0000 UTC m=+1862.738644257" watchObservedRunningTime="2026-09-16 12:32:53.10306321 +0000 UTC m=+1862.740388047"
Sep 16 12:32:53.442245 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:53.442164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:53.448601 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:53.448570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:54.088898 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:54.088857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:54.089805 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:54.089782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:32:57.552763 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.552727    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"]
Sep 16 12:32:57.555972 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.555948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.558367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.558343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3863e0-serving-cert\""
Sep 16 12:32:57.566432 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.566407    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"]
Sep 16 12:32:57.711043 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.711005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrvr\" (UniqueName: \"kubernetes.io/projected/3cfc88a1-f809-43c0-ad81-7a57718954de-kube-api-access-6wrvr\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.711227 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.711071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-metrics-certs\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.711227 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.711111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfc88a1-f809-43c0-ad81-7a57718954de-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.711227 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.711140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.812603 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.812502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-metrics-certs\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.812603 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.812567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfc88a1-f809-43c0-ad81-7a57718954de-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.812603 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.812599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.812804 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.812628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrvr\" (UniqueName: \"kubernetes.io/projected/3cfc88a1-f809-43c0-ad81-7a57718954de-kube-api-access-6wrvr\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.812804 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:32:57.812743    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-serving-cert: secret "maas-api-e2e-ipp-b-3863e0-serving-cert" not found
Sep 16 12:32:57.812887 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:32:57.812812    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls podName:3cfc88a1-f809-43c0-ad81-7a57718954de nodeName:}" failed. No retries permitted until 2026-09-16 12:32:58.312790326 +0000 UTC m=+1867.950115146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls") pod "maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55" (UID: "3cfc88a1-f809-43c0-ad81-7a57718954de") : secret "maas-api-e2e-ipp-b-3863e0-serving-cert" not found
Sep 16 12:32:57.813292 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.813266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfc88a1-f809-43c0-ad81-7a57718954de-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.815172 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.815153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-metrics-certs\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:57.823459 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:57.823442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrvr\" (UniqueName: \"kubernetes.io/projected/3cfc88a1-f809-43c0-ad81-7a57718954de-kube-api-access-6wrvr\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:58.317915 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:58.317878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:58.320351 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:58.320323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:58.468563 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:58.468535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:32:58.742195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:58.742169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"]
Sep 16 12:32:58.743311 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:32:58.743285    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfc88a1_f809_43c0_ad81_7a57718954de.slice/crio-0afd4330bd8c0d276762d160a9ab5e9633953c174ca84a0e0caf9bb4d7c8ffc6 WatchSource:0}: Error finding container 0afd4330bd8c0d276762d160a9ab5e9633953c174ca84a0e0caf9bb4d7c8ffc6: Status 404 returned error can't find the container with id 0afd4330bd8c0d276762d160a9ab5e9633953c174ca84a0e0caf9bb4d7c8ffc6
Sep 16 12:32:59.114285 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:32:59.114209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55" event={"ID":"3cfc88a1-f809-43c0-ad81-7a57718954de","Type":"ContainerStarted","Data":"0afd4330bd8c0d276762d160a9ab5e9633953c174ca84a0e0caf9bb4d7c8ffc6"}
Sep 16 12:33:00.121045 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:00.121013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55" event={"ID":"3cfc88a1-f809-43c0-ad81-7a57718954de","Type":"ContainerStarted","Data":"d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd"}
Sep 16 12:33:00.121451 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:00.121150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:33:00.142917 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:00.142873    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55" podStartSLOduration=2.691246872 podStartE2EDuration="3.142859343s" totalImagesPullingTime="451.612471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:33:00.1401939 +0000 UTC m=+1869.777518739" watchObservedRunningTime="2026-09-16 12:33:00.142859343 +0000 UTC m=+1869.780184181"
Sep 16 12:33:06.132540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:06.132484    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:33:10.130771 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:10.130711    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:33:10.131308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:10.130794    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:33:12.193130 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:12.193019    2578 generic.go:356] "Generic (PLEG): container finished" podID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerID="8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262" exitCode=0
Sep 16 12:33:12.193130 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:12.193074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" event={"ID":"776542f7-9926-46eb-94ca-21b473d3f7bc","Type":"ContainerDied","Data":"8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262"}
Sep 16 12:33:12.193624 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:12.193433    2578 scope.go:117] "RemoveContainer" containerID="8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262"
Sep 16 12:33:12.194457 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:12.194428    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:33:12.194524 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:12.194496    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:33:12.194571 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:12.194548    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 12:33:13.200866 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:13.200825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" event={"ID":"776542f7-9926-46eb-94ca-21b473d3f7bc","Type":"ContainerStarted","Data":"a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b"}
Sep 16 12:33:13.442012 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:13.441968    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:33:13.442374 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:13.442344    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 16 12:33:13.442458 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:13.442403    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 16 12:33:14.441712 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:14.441674    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 16 12:33:14.442308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:14.442274    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 16 12:33:15.441478 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:15.441432    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 16 12:33:15.441701 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:15.441501    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 16 12:33:16.441756 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:16.441710    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 16 12:33:16.442263 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:16.441782    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 16 12:33:17.442091 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:17.442041    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 16 12:33:17.442633 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:17.442117    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 16 12:33:18.444988 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:18.444956    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:33:18.445469 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:18.445266    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:33:18.446025 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:18.446008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:33:42.961528 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:42.961477    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"]
Sep 16 12:33:42.964119 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:42.964100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:42.966987 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:42.966963    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a4f954\"/\"openshift-service-ca.crt\""
Sep 16 12:33:42.968024 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:42.968007    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a4f954\"/\"kube-root-ca.crt\""
Sep 16 12:33:42.968110 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:42.968015    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a4f954\"/\"ipp-route-a4f954-kserve-self-signed-certs\""
Sep 16 12:33:42.968253 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:42.968137    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a4f954\"/\"default-dockercfg-h7fzx\""
Sep 16 12:33:42.974272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:42.974247    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"]
Sep 16 12:33:43.124889 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.124853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-dshm\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.125109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.124913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-home\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.125109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.124997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-model-cache\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.125109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.125071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tls-certs\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.125109 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.125104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748xx\" (UniqueName: \"kubernetes.io/projected/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-kube-api-access-748xx\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.125302 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.125142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tmp-dir\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226049 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.225964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-dshm\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226049 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.226017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-home\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226049 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.226051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-model-cache\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226357 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.226072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tls-certs\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226357 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.226092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748xx\" (UniqueName: \"kubernetes.io/projected/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-kube-api-access-748xx\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226357 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.226115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tmp-dir\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226552 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.226475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-model-cache\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226621 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.226542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-home\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.226695 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.226675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tmp-dir\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.228378 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.228356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-dshm\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.228722 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.228706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tls-certs\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.235786 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.235764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-748xx\" (UniqueName: \"kubernetes.io/projected/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-kube-api-access-748xx\") pod \"ipp-route-a4f954-kserve-cd9f4c7f5-blbpf\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") " pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.273789 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.273758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:43.558374 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:43.558349    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"]
Sep 16 12:33:43.559935 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:33:43.559908    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9da3cdc_9e4b_4b4c_85b4_f4826adad433.slice/crio-7d3148a650ff9a826d7c7da23459bdb3ba5971f119308324d38ae3bff055f783 WatchSource:0}: Error finding container 7d3148a650ff9a826d7c7da23459bdb3ba5971f119308324d38ae3bff055f783: Status 404 returned error can't find the container with id 7d3148a650ff9a826d7c7da23459bdb3ba5971f119308324d38ae3bff055f783
Sep 16 12:33:44.372871 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:44.372832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf" event={"ID":"c9da3cdc-9e4b-4b4c-85b4-f4826adad433","Type":"ContainerStarted","Data":"0bee616ae54c8477e9d2ce2cc026bd241276f424d2c8e0adfaff9be88a85d7fe"}
Sep 16 12:33:44.373268 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:44.372884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf" event={"ID":"c9da3cdc-9e4b-4b4c-85b4-f4826adad433","Type":"ContainerStarted","Data":"7d3148a650ff9a826d7c7da23459bdb3ba5971f119308324d38ae3bff055f783"}
Sep 16 12:33:44.391986 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:44.391935    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf" podStartSLOduration=2.3919190390000002 podStartE2EDuration="2.391919039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:33:44.389222835 +0000 UTC m=+1914.026547671" watchObservedRunningTime="2026-09-16 12:33:44.391919039 +0000 UTC m=+1914.029243877"
Sep 16 12:33:53.274579 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:53.274542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:53.274970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:53.274599    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:53.287283 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:53.287260    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:33:53.430334 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:33:53.430300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:34:19.558564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.558529    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"]
Sep 16 12:34:19.563312 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.563294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.566022 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.565994    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-555bf4-serving-cert\""
Sep 16 12:34:19.573334 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.573307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"]
Sep 16 12:34:19.648692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.648661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrjf\" (UniqueName: \"kubernetes.io/projected/cb3f787a-9fb2-4413-9325-4becd20ddf6d-kube-api-access-tqrjf\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.648861 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.648720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3f787a-9fb2-4413-9325-4becd20ddf6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.648861 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.648801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.648861 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.648821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.750118 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.750084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrjf\" (UniqueName: \"kubernetes.io/projected/cb3f787a-9fb2-4413-9325-4becd20ddf6d-kube-api-access-tqrjf\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.750327 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.750145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3f787a-9fb2-4413-9325-4becd20ddf6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.750327 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.750192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.750327 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.750210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.750327 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:34:19.750300    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-serving-cert: secret "maas-api-e2e-ipp-cleanup-555bf4-serving-cert" not found
Sep 16 12:34:19.750568 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:34:19.750360    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls podName:cb3f787a-9fb2-4413-9325-4becd20ddf6d nodeName:}" failed. No retries permitted until 2026-09-16 12:34:20.250343123 +0000 UTC m=+1949.887667952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87" (UID: "cb3f787a-9fb2-4413-9325-4becd20ddf6d") : secret "maas-api-e2e-ipp-cleanup-555bf4-serving-cert" not found
Sep 16 12:34:19.750827 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.750800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3f787a-9fb2-4413-9325-4becd20ddf6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.752806 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.752788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:19.761384 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:19.761357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrjf\" (UniqueName: \"kubernetes.io/projected/cb3f787a-9fb2-4413-9325-4becd20ddf6d-kube-api-access-tqrjf\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:20.255482 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:20.254925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:20.259540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:20.258491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:20.473900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:20.473868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:20.746359 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:20.746329    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"]
Sep 16 12:34:20.747731 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:34:20.747704    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3f787a_9fb2_4413_9325_4becd20ddf6d.slice/crio-6f583b1dd33bd396a99e2ce006137a826568aa718844e3358920c03d108d04a9 WatchSource:0}: Error finding container 6f583b1dd33bd396a99e2ce006137a826568aa718844e3358920c03d108d04a9: Status 404 returned error can't find the container with id 6f583b1dd33bd396a99e2ce006137a826568aa718844e3358920c03d108d04a9
Sep 16 12:34:21.568621 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:21.568580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87" event={"ID":"cb3f787a-9fb2-4413-9325-4becd20ddf6d","Type":"ContainerStarted","Data":"e39243bba493596b77222211b05edbb7d0745174fcd4bad5061b683537067346"}
Sep 16 12:34:21.568819 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:21.568636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87" event={"ID":"cb3f787a-9fb2-4413-9325-4becd20ddf6d","Type":"ContainerStarted","Data":"6f583b1dd33bd396a99e2ce006137a826568aa718844e3358920c03d108d04a9"}
Sep 16 12:34:21.568819 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:21.568757    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:21.587004 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:21.586955    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87" podStartSLOduration=2.181203459 podStartE2EDuration="2.586939822s" totalImagesPullingTime="405.736363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:34:21.585247884 +0000 UTC m=+1951.222572761" watchObservedRunningTime="2026-09-16 12:34:21.586939822 +0000 UTC m=+1951.224264660"
Sep 16 12:34:27.579563 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:27.579533    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:57.624186 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.624106    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"]
Sep 16 12:34:57.624769 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.624377    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87" podUID="cb3f787a-9fb2-4413-9325-4becd20ddf6d" containerName="maas-api" containerID="cri-o://e39243bba493596b77222211b05edbb7d0745174fcd4bad5061b683537067346" gracePeriod=30
Sep 16 12:34:57.773501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.773464    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb3f787a-9fb2-4413-9325-4becd20ddf6d" containerID="e39243bba493596b77222211b05edbb7d0745174fcd4bad5061b683537067346" exitCode=0
Sep 16 12:34:57.773671 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.773548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87" event={"ID":"cb3f787a-9fb2-4413-9325-4becd20ddf6d","Type":"ContainerDied","Data":"e39243bba493596b77222211b05edbb7d0745174fcd4bad5061b683537067346"}
Sep 16 12:34:57.873791 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.873768    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:57.974352 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.974313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls\") pod \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") "
Sep 16 12:34:57.974564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.974408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqrjf\" (UniqueName: \"kubernetes.io/projected/cb3f787a-9fb2-4413-9325-4becd20ddf6d-kube-api-access-tqrjf\") pod \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") "
Sep 16 12:34:57.974564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.974425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3f787a-9fb2-4413-9325-4becd20ddf6d-openshift-service-ca\") pod \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") "
Sep 16 12:34:57.974564 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.974452    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-metrics-certs\") pod \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\" (UID: \"cb3f787a-9fb2-4413-9325-4becd20ddf6d\") "
Sep 16 12:34:57.974876 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.974846    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3f787a-9fb2-4413-9325-4becd20ddf6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb3f787a-9fb2-4413-9325-4becd20ddf6d" (UID: "cb3f787a-9fb2-4413-9325-4becd20ddf6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:34:57.975168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.975148    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3f787a-9fb2-4413-9325-4becd20ddf6d-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:34:57.976780 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.976756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb3f787a-9fb2-4413-9325-4becd20ddf6d" (UID: "cb3f787a-9fb2-4413-9325-4becd20ddf6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:34:57.977208 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.977189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb3f787a-9fb2-4413-9325-4becd20ddf6d" (UID: "cb3f787a-9fb2-4413-9325-4becd20ddf6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:34:57.977264 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:57.977223    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3f787a-9fb2-4413-9325-4becd20ddf6d-kube-api-access-tqrjf" (OuterVolumeSpecName: "kube-api-access-tqrjf") pod "cb3f787a-9fb2-4413-9325-4becd20ddf6d" (UID: "cb3f787a-9fb2-4413-9325-4becd20ddf6d"). InnerVolumeSpecName "kube-api-access-tqrjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:58.076101 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.076060    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqrjf\" (UniqueName: \"kubernetes.io/projected/cb3f787a-9fb2-4413-9325-4becd20ddf6d-kube-api-access-tqrjf\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:34:58.076101 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.076097    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:34:58.076101 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.076107    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3f787a-9fb2-4413-9325-4becd20ddf6d-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:34:58.780710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.780682    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"
Sep 16 12:34:58.780710 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.780697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87" event={"ID":"cb3f787a-9fb2-4413-9325-4becd20ddf6d","Type":"ContainerDied","Data":"6f583b1dd33bd396a99e2ce006137a826568aa718844e3358920c03d108d04a9"}
Sep 16 12:34:58.781225 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.780788    2578 scope.go:117] "RemoveContainer" containerID="e39243bba493596b77222211b05edbb7d0745174fcd4bad5061b683537067346"
Sep 16 12:34:58.803242 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.803207    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"]
Sep 16 12:34:58.807121 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.807098    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-555bf4-569f54fb45-5pk87"]
Sep 16 12:34:58.884911 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:34:58.884888    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3f787a-9fb2-4413-9325-4becd20ddf6d" path="/var/lib/kubelet/pods/cb3f787a-9fb2-4413-9325-4becd20ddf6d/volumes"
Sep 16 12:35:24.266380 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.266333    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"]
Sep 16 12:35:24.267003 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.266573    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb" podUID="d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" containerName="maas-api" containerID="cri-o://2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423" gracePeriod=30
Sep 16 12:35:24.518322 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.518252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:35:24.612245 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.612209    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-openshift-service-ca\") pod \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") "
Sep 16 12:35:24.612428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.612276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqjzz\" (UniqueName: \"kubernetes.io/projected/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-kube-api-access-vqjzz\") pod \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") "
Sep 16 12:35:24.612428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.612294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-metrics-certs\") pod \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") "
Sep 16 12:35:24.612428 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.612325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls\") pod \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\" (UID: \"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff\") "
Sep 16 12:35:24.612683 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.612648    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" (UID: "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:35:24.614628 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.614599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" (UID: "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:35:24.614628 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.614617    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" (UID: "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:35:24.614766 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.614640    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-kube-api-access-vqjzz" (OuterVolumeSpecName: "kube-api-access-vqjzz") pod "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" (UID: "d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff"). InnerVolumeSpecName "kube-api-access-vqjzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:24.713337 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.713307    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqjzz\" (UniqueName: \"kubernetes.io/projected/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-kube-api-access-vqjzz\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:24.713337 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.713333    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:24.713337 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.713341    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:24.713574 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.713349    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:24.917819 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.917786    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" containerID="2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423" exitCode=0
Sep 16 12:35:24.917959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.917830    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"
Sep 16 12:35:24.917959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.917867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb" event={"ID":"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff","Type":"ContainerDied","Data":"2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423"}
Sep 16 12:35:24.917959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.917920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb" event={"ID":"d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff","Type":"ContainerDied","Data":"fff2abf4122a257051aa7e71fe23f5f8c9a1b408684d397aa19d072fca4712de"}
Sep 16 12:35:24.917959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.917949    2578 scope.go:117] "RemoveContainer" containerID="2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423"
Sep 16 12:35:24.927314 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.927288    2578 scope.go:117] "RemoveContainer" containerID="2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423"
Sep 16 12:35:24.927591 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:35:24.927573    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423\": container with ID starting with 2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423 not found: ID does not exist" containerID="2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423"
Sep 16 12:35:24.927643 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.927598    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423"} err="failed to get container status \"2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423\": rpc error: code = NotFound desc = could not find container \"2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423\": container with ID starting with 2f368984f1e82ca59a11b3e38e3b712b64876f14288c7a43290c2161e15a9423 not found: ID does not exist"
Sep 16 12:35:24.937139 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.937117    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"]
Sep 16 12:35:24.945945 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:24.945924    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a4f954-55d6cc6b76-bjxgb"]
Sep 16 12:35:26.885057 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:26.885014    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" path="/var/lib/kubelet/pods/d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff/volumes"
Sep 16 12:35:29.872067 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:29.872032    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"]
Sep 16 12:35:44.891638 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:44.891595    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf" podUID="c9da3cdc-9e4b-4b4c-85b4-f4826adad433" containerName="main" containerID="cri-o://0bee616ae54c8477e9d2ce2cc026bd241276f424d2c8e0adfaff9be88a85d7fe" gracePeriod=45
Sep 16 12:35:45.029174 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.029141    2578 generic.go:356] "Generic (PLEG): container finished" podID="c9da3cdc-9e4b-4b4c-85b4-f4826adad433" containerID="0bee616ae54c8477e9d2ce2cc026bd241276f424d2c8e0adfaff9be88a85d7fe" exitCode=0
Sep 16 12:35:45.029340 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.029217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf" event={"ID":"c9da3cdc-9e4b-4b4c-85b4-f4826adad433","Type":"ContainerDied","Data":"0bee616ae54c8477e9d2ce2cc026bd241276f424d2c8e0adfaff9be88a85d7fe"}
Sep 16 12:35:45.131085 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.131056    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:35:45.206101 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-dshm\") pod \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") "
Sep 16 12:35:45.206274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-model-cache\") pod \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") "
Sep 16 12:35:45.206274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tls-certs\") pod \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") "
Sep 16 12:35:45.206274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206186    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tmp-dir\") pod \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") "
Sep 16 12:35:45.206274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-home\") pod \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") "
Sep 16 12:35:45.206504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-748xx\" (UniqueName: \"kubernetes.io/projected/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-kube-api-access-748xx\") pod \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\" (UID: \"c9da3cdc-9e4b-4b4c-85b4-f4826adad433\") "
Sep 16 12:35:45.206504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9da3cdc-9e4b-4b4c-85b4-f4826adad433" (UID: "c9da3cdc-9e4b-4b4c-85b4-f4826adad433"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:45.206644 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206508    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-home" (OuterVolumeSpecName: "home") pod "c9da3cdc-9e4b-4b4c-85b4-f4826adad433" (UID: "c9da3cdc-9e4b-4b4c-85b4-f4826adad433"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:45.206644 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206535    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9da3cdc-9e4b-4b4c-85b4-f4826adad433" (UID: "c9da3cdc-9e4b-4b4c-85b4-f4826adad433"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:45.206907 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206887    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:45.206955 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206914    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:45.206955 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.206927    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:45.208418 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.208394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-dshm" (OuterVolumeSpecName: "dshm") pod "c9da3cdc-9e4b-4b4c-85b4-f4826adad433" (UID: "c9da3cdc-9e4b-4b4c-85b4-f4826adad433"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:45.208540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.208412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-kube-api-access-748xx" (OuterVolumeSpecName: "kube-api-access-748xx") pod "c9da3cdc-9e4b-4b4c-85b4-f4826adad433" (UID: "c9da3cdc-9e4b-4b4c-85b4-f4826adad433"). InnerVolumeSpecName "kube-api-access-748xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:45.208540 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.208424    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9da3cdc-9e4b-4b4c-85b4-f4826adad433" (UID: "c9da3cdc-9e4b-4b4c-85b4-f4826adad433"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:35:45.307854 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.307826    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-748xx\" (UniqueName: \"kubernetes.io/projected/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-kube-api-access-748xx\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:45.307970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.307857    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:45.307970 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:45.307869    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9da3cdc-9e4b-4b4c-85b4-f4826adad433-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:35:46.036121 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:46.036094    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"
Sep 16 12:35:46.036672 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:46.036090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf" event={"ID":"c9da3cdc-9e4b-4b4c-85b4-f4826adad433","Type":"ContainerDied","Data":"7d3148a650ff9a826d7c7da23459bdb3ba5971f119308324d38ae3bff055f783"}
Sep 16 12:35:46.036672 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:46.036244    2578 scope.go:117] "RemoveContainer" containerID="0bee616ae54c8477e9d2ce2cc026bd241276f424d2c8e0adfaff9be88a85d7fe"
Sep 16 12:35:46.056751 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:46.056726    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"]
Sep 16 12:35:46.060717 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:46.060697    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a4f954/ipp-route-a4f954-kserve-cd9f4c7f5-blbpf"]
Sep 16 12:35:46.884219 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:35:46.884184    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9da3cdc-9e4b-4b4c-85b4-f4826adad433" path="/var/lib/kubelet/pods/c9da3cdc-9e4b-4b4c-85b4-f4826adad433/volumes"
Sep 16 12:36:23.735730 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:23.735638    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"]
Sep 16 12:36:23.736327 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:23.735930    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55" podUID="3cfc88a1-f809-43c0-ad81-7a57718954de" containerName="maas-api" containerID="cri-o://d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd" gracePeriod=30
Sep 16 12:36:23.980195 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:23.980170    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:36:24.145462 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.145372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls\") pod \"3cfc88a1-f809-43c0-ad81-7a57718954de\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") "
Sep 16 12:36:24.145462 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.145424    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-metrics-certs\") pod \"3cfc88a1-f809-43c0-ad81-7a57718954de\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") "
Sep 16 12:36:24.145792 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.145528    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wrvr\" (UniqueName: \"kubernetes.io/projected/3cfc88a1-f809-43c0-ad81-7a57718954de-kube-api-access-6wrvr\") pod \"3cfc88a1-f809-43c0-ad81-7a57718954de\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") "
Sep 16 12:36:24.145792 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.145586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfc88a1-f809-43c0-ad81-7a57718954de-openshift-service-ca\") pod \"3cfc88a1-f809-43c0-ad81-7a57718954de\" (UID: \"3cfc88a1-f809-43c0-ad81-7a57718954de\") "
Sep 16 12:36:24.146003 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.145977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfc88a1-f809-43c0-ad81-7a57718954de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cfc88a1-f809-43c0-ad81-7a57718954de" (UID: "3cfc88a1-f809-43c0-ad81-7a57718954de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:36:24.147729 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.147700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3cfc88a1-f809-43c0-ad81-7a57718954de" (UID: "3cfc88a1-f809-43c0-ad81-7a57718954de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:36:24.147862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.147832    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfc88a1-f809-43c0-ad81-7a57718954de-kube-api-access-6wrvr" (OuterVolumeSpecName: "kube-api-access-6wrvr") pod "3cfc88a1-f809-43c0-ad81-7a57718954de" (UID: "3cfc88a1-f809-43c0-ad81-7a57718954de"). InnerVolumeSpecName "kube-api-access-6wrvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:36:24.147926 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.147863    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cfc88a1-f809-43c0-ad81-7a57718954de" (UID: "3cfc88a1-f809-43c0-ad81-7a57718954de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:36:24.240375 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.240343    2578 generic.go:356] "Generic (PLEG): container finished" podID="3cfc88a1-f809-43c0-ad81-7a57718954de" containerID="d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd" exitCode=0
Sep 16 12:36:24.240551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.240385    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"
Sep 16 12:36:24.240551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.240422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55" event={"ID":"3cfc88a1-f809-43c0-ad81-7a57718954de","Type":"ContainerDied","Data":"d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd"}
Sep 16 12:36:24.240551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.240478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55" event={"ID":"3cfc88a1-f809-43c0-ad81-7a57718954de","Type":"ContainerDied","Data":"0afd4330bd8c0d276762d160a9ab5e9633953c174ca84a0e0caf9bb4d7c8ffc6"}
Sep 16 12:36:24.240551 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.240504    2578 scope.go:117] "RemoveContainer" containerID="d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd"
Sep 16 12:36:24.246855 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.246832    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:24.246940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.246861    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfc88a1-f809-43c0-ad81-7a57718954de-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:24.246940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.246877    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wrvr\" (UniqueName: \"kubernetes.io/projected/3cfc88a1-f809-43c0-ad81-7a57718954de-kube-api-access-6wrvr\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:24.246940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.246891    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfc88a1-f809-43c0-ad81-7a57718954de-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:24.250353 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.250338    2578 scope.go:117] "RemoveContainer" containerID="d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd"
Sep 16 12:36:24.250615 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:36:24.250597    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd\": container with ID starting with d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd not found: ID does not exist" containerID="d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd"
Sep 16 12:36:24.250670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.250623    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd"} err="failed to get container status \"d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd\": rpc error: code = NotFound desc = could not find container \"d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd\": container with ID starting with d0b8a6598bbba647ebb4ab50a61c8a53ca2f72f3f124f8e1b1843ab139318ddd not found: ID does not exist"
Sep 16 12:36:24.261919 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.261892    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"]
Sep 16 12:36:24.265943 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.265921    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3863e0-5c8df68cd5-9lk55"]
Sep 16 12:36:24.884725 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:24.884687    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cfc88a1-f809-43c0-ad81-7a57718954de" path="/var/lib/kubelet/pods/3cfc88a1-f809-43c0-ad81-7a57718954de/volumes"
Sep 16 12:36:30.607592 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:30.607559    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"]
Sep 16 12:36:30.608108 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:30.607847    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy" containerID="cri-o://a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b" gracePeriod=30
Sep 16 12:36:33.447830 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:33.447791    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:36:33.448200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:33.447864    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:36:35.852832 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.852809    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:36:35.947640 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-credential-socket\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.947640 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-envoy\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.947640 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947634    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-socket\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.947892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947686    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776542f7-9926-46eb-94ca-21b473d3f7bc-istiod-ca-cert\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.947892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-certs\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.947892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-data\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.947892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-token\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.947892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffrkj\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-kube-api-access-ffrkj\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.947892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947848    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:35.947892 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-podinfo\") pod \"776542f7-9926-46eb-94ca-21b473d3f7bc\" (UID: \"776542f7-9926-46eb-94ca-21b473d3f7bc\") "
Sep 16 12:36:35.948240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.947950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:35.948240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.948083    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:35.948240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.948145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776542f7-9926-46eb-94ca-21b473d3f7bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:36:35.948240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.948222    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776542f7-9926-46eb-94ca-21b473d3f7bc-istiod-ca-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:35.948240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.948235    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:35.948240 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.948244    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-credential-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:35.948559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.948255    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-workload-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:35.948559 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.948450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:35.950051 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.950027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:35.950201 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.950170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:36:35.950591 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.950570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:36:35.950692 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:35.950586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-kube-api-access-ffrkj" (OuterVolumeSpecName: "kube-api-access-ffrkj") pod "776542f7-9926-46eb-94ca-21b473d3f7bc" (UID: "776542f7-9926-46eb-94ca-21b473d3f7bc"). InnerVolumeSpecName "kube-api-access-ffrkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:36:36.049670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.049635    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-podinfo\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:36.049670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.049664    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-envoy\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:36.049670 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.049673    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-data\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:36.049907 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.049682    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-istio-token\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:36.049907 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.049691    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffrkj\" (UniqueName: \"kubernetes.io/projected/776542f7-9926-46eb-94ca-21b473d3f7bc-kube-api-access-ffrkj\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:36.307122 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.307091    2578 generic.go:356] "Generic (PLEG): container finished" podID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerID="a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b" exitCode=0
Sep 16 12:36:36.307308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.307156    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"
Sep 16 12:36:36.307308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.307178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" event={"ID":"776542f7-9926-46eb-94ca-21b473d3f7bc","Type":"ContainerDied","Data":"a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b"}
Sep 16 12:36:36.307308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.307236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb" event={"ID":"776542f7-9926-46eb-94ca-21b473d3f7bc","Type":"ContainerDied","Data":"ca039cc95e367d965fdecb7babc437b6dc5987bc69614f58eef8180dfc85a02f"}
Sep 16 12:36:36.307308 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.307264    2578 scope.go:117] "RemoveContainer" containerID="a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b"
Sep 16 12:36:36.317470 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.317449    2578 scope.go:117] "RemoveContainer" containerID="8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262"
Sep 16 12:36:36.325636 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.325617    2578 scope.go:117] "RemoveContainer" containerID="a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b"
Sep 16 12:36:36.325887 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:36:36.325868    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b\": container with ID starting with a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b not found: ID does not exist" containerID="a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b"
Sep 16 12:36:36.325935 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.325895    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b"} err="failed to get container status \"a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b\": rpc error: code = NotFound desc = could not find container \"a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b\": container with ID starting with a654dda8dd6767f0ad7f51a0b39c001784b0fd22efa53a197896a96b31d9323b not found: ID does not exist"
Sep 16 12:36:36.325935 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.325912    2578 scope.go:117] "RemoveContainer" containerID="8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262"
Sep 16 12:36:36.326212 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:36:36.326193    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262\": container with ID starting with 8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262 not found: ID does not exist" containerID="8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262"
Sep 16 12:36:36.326269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.326217    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262"} err="failed to get container status \"8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262\": rpc error: code = NotFound desc = could not find container \"8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262\": container with ID starting with 8ffbc710b456425e1eeb5df7829f8816b018663df9b3a853ca66205a55996262 not found: ID does not exist"
Sep 16 12:36:36.348151 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.348128    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"]
Sep 16 12:36:36.354850 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.354830    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3863e0-openshift-default-78858b6598-jg9xb"]
Sep 16 12:36:36.885175 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:36.885145    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" path="/var/lib/kubelet/pods/776542f7-9926-46eb-94ca-21b473d3f7bc/volumes"
Sep 16 12:36:48.759323 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:48.759286    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"]
Sep 16 12:36:48.759848 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:48.759507    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy" containerID="cri-o://f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3" gracePeriod=30
Sep 16 12:36:51.036475 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:51.036453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:36:51.046048 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:51.046027    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:36:53.149986 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:53.149946    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:36:53.150555 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:53.150044    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:36:54.020425 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.020401    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:36:54.105828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.105743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-envoy\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.105828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.105783    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-token\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.105828 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.105816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-data\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.106118 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.105837    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-podinfo\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.106118 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.105927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-credential-socket\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.106118 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.105964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-socket\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.106118 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106027    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istiod-ca-cert\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.106118 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106081    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-certs\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.106118 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pl2d\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-kube-api-access-4pl2d\") pod \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\" (UID: \"63859a7d-a1da-45dd-abc0-95bdd0023b5c\") "
Sep 16 12:36:54.106434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:54.106434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:54.106434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106385    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:54.106434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106395    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-credential-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.106434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106413    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-data\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.106434 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:36:54.106952 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.106756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:54.108160 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.108128    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:36:54.108294 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.108199    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:36:54.108501 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.108482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:36:54.108611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.108595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-kube-api-access-4pl2d" (OuterVolumeSpecName: "kube-api-access-4pl2d") pod "63859a7d-a1da-45dd-abc0-95bdd0023b5c" (UID: "63859a7d-a1da-45dd-abc0-95bdd0023b5c"). InnerVolumeSpecName "kube-api-access-4pl2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:36:54.207343 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.207310    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istiod-ca-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.207343 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.207336    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.207343 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.207345    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl2d\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-kube-api-access-4pl2d\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.207343 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.207353    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-envoy\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.207843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.207362    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-token\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.207843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.207369    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63859a7d-a1da-45dd-abc0-95bdd0023b5c-istio-podinfo\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.207843 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.207375    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63859a7d-a1da-45dd-abc0-95bdd0023b5c-workload-socket\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:36:54.409936 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.409849    2578 generic.go:356] "Generic (PLEG): container finished" podID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerID="f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3" exitCode=0
Sep 16 12:36:54.409936 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.409908    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"
Sep 16 12:36:54.409936 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.409909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" event={"ID":"63859a7d-a1da-45dd-abc0-95bdd0023b5c","Type":"ContainerDied","Data":"f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3"}
Sep 16 12:36:54.410154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.409987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p" event={"ID":"63859a7d-a1da-45dd-abc0-95bdd0023b5c","Type":"ContainerDied","Data":"9537e4db54c54c651419060dc6a909a5b8ac3cb1863ec72b06a2bfc747e84510"}
Sep 16 12:36:54.410154 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.410010    2578 scope.go:117] "RemoveContainer" containerID="f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3"
Sep 16 12:36:54.422395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.422370    2578 scope.go:117] "RemoveContainer" containerID="4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb"
Sep 16 12:36:54.430717 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.430693    2578 scope.go:117] "RemoveContainer" containerID="f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3"
Sep 16 12:36:54.431166 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:36:54.431144    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3\": container with ID starting with f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3 not found: ID does not exist" containerID="f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3"
Sep 16 12:36:54.431248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.431172    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3"} err="failed to get container status \"f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3\": rpc error: code = NotFound desc = could not find container \"f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3\": container with ID starting with f2e421d63452f0a609fc2332e190d8781983fbced5ea9507899b27aaa993ecd3 not found: ID does not exist"
Sep 16 12:36:54.431248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.431189    2578 scope.go:117] "RemoveContainer" containerID="4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb"
Sep 16 12:36:54.431508 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:36:54.431488    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb\": container with ID starting with 4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb not found: ID does not exist" containerID="4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb"
Sep 16 12:36:54.431619 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.431573    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb"} err="failed to get container status \"4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb\": rpc error: code = NotFound desc = could not find container \"4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb\": container with ID starting with 4bf3a7fa45d28b243c08c24b45df354eb1a9efe60fe2081215887003696b61eb not found: ID does not exist"
Sep 16 12:36:54.433269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.433249    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"]
Sep 16 12:36:54.436430 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.436411    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-db42a6-openshift-default-786d8d4b4-t4s4p"]
Sep 16 12:36:54.885402 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:54.885363    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" path="/var/lib/kubelet/pods/63859a7d-a1da-45dd-abc0-95bdd0023b5c/volumes"
Sep 16 12:36:59.683711 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:59.683668    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"]
Sep 16 12:36:59.685882 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:36:59.685859    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c03953-29826030-zrhjh"]
Sep 16 12:37:00.885993 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:37:00.885957    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="258f9816-044c-418b-8575-7d6893f8269f" path="/var/lib/kubelet/pods/258f9816-044c-418b-8575-7d6893f8269f/volumes"
Sep 16 12:37:51.282273 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:37:51.282192    2578 scope.go:117] "RemoveContainer" containerID="5c5d5063f5a82479a9255cb136fdba5660e015a36f2f93e3f232b57c7e5f9449"
Sep 16 12:38:13.754705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.754672    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d8575c494-dgkbx"]
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755009    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755021    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755029    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3f787a-9fb2-4413-9325-4becd20ddf6d" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755035    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3f787a-9fb2-4413-9325-4becd20ddf6d" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755045    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755051    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755058    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755063    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755076    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cfc88a1-f809-43c0-ad81-7a57718954de" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755081    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cfc88a1-f809-43c0-ad81-7a57718954de" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755089    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755094    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755102    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9da3cdc-9e4b-4b4c-85b4-f4826adad433" containerName="main"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755107    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9da3cdc-9e4b-4b4c-85b4-f4826adad433" containerName="main"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755160    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755167    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3f787a-9fb2-4413-9325-4becd20ddf6d" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755173    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755179    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9da3cdc-9e4b-4b4c-85b4-f4826adad433" containerName="main"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755184    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f8f74b-8706-4bd2-8b0c-7a56fdc99dff" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755189    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cfc88a1-f809-43c0-ad81-7a57718954de" containerName="maas-api"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755196    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy"
Sep 16 12:38:13.756920 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.755201    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="63859a7d-a1da-45dd-abc0-95bdd0023b5c" containerName="istio-proxy"
Sep 16 12:38:13.758006 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.757991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.762568 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.762541    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 12:38:13.762703 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.762591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 12:38:13.762703 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.762612    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddsz6\""
Sep 16 12:38:13.762703 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.762620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 12:38:13.768972 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.768945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d8575c494-dgkbx"]
Sep 16 12:38:13.868560 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.868530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-webhook-cert\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.868726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.868622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-metrics-certs\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.868726 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.868645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pst4l\" (UniqueName: \"kubernetes.io/projected/76a0cc0b-da61-41fb-b849-ce7593a81023-kube-api-access-pst4l\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.969851 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.969818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-webhook-cert\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.969999 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.969881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-metrics-certs\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.969999 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.969897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pst4l\" (UniqueName: \"kubernetes.io/projected/76a0cc0b-da61-41fb-b849-ce7593a81023-kube-api-access-pst4l\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.972445 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.972417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-metrics-certs\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.972572 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.972451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-webhook-cert\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:13.978321 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:13.978302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pst4l\" (UniqueName: \"kubernetes.io/projected/76a0cc0b-da61-41fb-b849-ce7593a81023-kube-api-access-pst4l\") pod \"maas-controller-d8575c494-dgkbx\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") " pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:14.070228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:14.070162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:14.341507 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:14.341418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d8575c494-dgkbx"]
Sep 16 12:38:14.344932 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:38:14.344901    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a0cc0b_da61_41fb_b849_ce7593a81023.slice/crio-e5ff1d91a9c4ac1b2b08cd2efc875e6afb4f149864e14f51aad3f1bf8a37372b WatchSource:0}: Error finding container e5ff1d91a9c4ac1b2b08cd2efc875e6afb4f149864e14f51aad3f1bf8a37372b: Status 404 returned error can't find the container with id e5ff1d91a9c4ac1b2b08cd2efc875e6afb4f149864e14f51aad3f1bf8a37372b
Sep 16 12:38:14.346281 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:14.346260    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:38:14.828033 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:14.828008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d8575c494-dgkbx" event={"ID":"76a0cc0b-da61-41fb-b849-ce7593a81023","Type":"ContainerStarted","Data":"e5ff1d91a9c4ac1b2b08cd2efc875e6afb4f149864e14f51aad3f1bf8a37372b"}
Sep 16 12:38:15.835259 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:15.835225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d8575c494-dgkbx" event={"ID":"76a0cc0b-da61-41fb-b849-ce7593a81023","Type":"ContainerStarted","Data":"6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253"}
Sep 16 12:38:15.835662 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:15.835327    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:15.855952 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:15.855908    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d8575c494-dgkbx" podStartSLOduration=2.404017997 podStartE2EDuration="2.855896778s" totalImagesPullingTime="451.878781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:15.852603717 +0000 UTC m=+2185.489928555" watchObservedRunningTime="2026-09-16 12:38:15.855896778 +0000 UTC m=+2185.493221617"
Sep 16 12:38:26.843855 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:26.843823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:36.188484 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.188437    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d8575c494-dgkbx"]
Sep 16 12:38:36.188938 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.188718    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d8575c494-dgkbx" podUID="76a0cc0b-da61-41fb-b849-ce7593a81023" containerName="manager" containerID="cri-o://6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253" gracePeriod=30
Sep 16 12:38:36.426969 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.426948    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:36.564087 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.564054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-webhook-cert\") pod \"76a0cc0b-da61-41fb-b849-ce7593a81023\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") "
Sep 16 12:38:36.564284 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.564103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-metrics-certs\") pod \"76a0cc0b-da61-41fb-b849-ce7593a81023\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") "
Sep 16 12:38:36.564284 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.564199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pst4l\" (UniqueName: \"kubernetes.io/projected/76a0cc0b-da61-41fb-b849-ce7593a81023-kube-api-access-pst4l\") pod \"76a0cc0b-da61-41fb-b849-ce7593a81023\" (UID: \"76a0cc0b-da61-41fb-b849-ce7593a81023\") "
Sep 16 12:38:36.566280 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.566246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "76a0cc0b-da61-41fb-b849-ce7593a81023" (UID: "76a0cc0b-da61-41fb-b849-ce7593a81023"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:38:36.566395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.566281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "76a0cc0b-da61-41fb-b849-ce7593a81023" (UID: "76a0cc0b-da61-41fb-b849-ce7593a81023"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:38:36.566395 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.566322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a0cc0b-da61-41fb-b849-ce7593a81023-kube-api-access-pst4l" (OuterVolumeSpecName: "kube-api-access-pst4l") pod "76a0cc0b-da61-41fb-b849-ce7593a81023" (UID: "76a0cc0b-da61-41fb-b849-ce7593a81023"). InnerVolumeSpecName "kube-api-access-pst4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:38:36.665252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.665216    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pst4l\" (UniqueName: \"kubernetes.io/projected/76a0cc0b-da61-41fb-b849-ce7593a81023-kube-api-access-pst4l\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:38:36.665252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.665245    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-webhook-cert\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:38:36.665252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.665255    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a0cc0b-da61-41fb-b849-ce7593a81023-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:38:36.946984 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.946954    2578 generic.go:356] "Generic (PLEG): container finished" podID="76a0cc0b-da61-41fb-b849-ce7593a81023" containerID="6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253" exitCode=0
Sep 16 12:38:36.947155 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.946997    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d8575c494-dgkbx"
Sep 16 12:38:36.947155 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.947043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d8575c494-dgkbx" event={"ID":"76a0cc0b-da61-41fb-b849-ce7593a81023","Type":"ContainerDied","Data":"6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253"}
Sep 16 12:38:36.947155 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.947090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d8575c494-dgkbx" event={"ID":"76a0cc0b-da61-41fb-b849-ce7593a81023","Type":"ContainerDied","Data":"e5ff1d91a9c4ac1b2b08cd2efc875e6afb4f149864e14f51aad3f1bf8a37372b"}
Sep 16 12:38:36.947155 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.947111    2578 scope.go:117] "RemoveContainer" containerID="6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253"
Sep 16 12:38:36.956225 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.956206    2578 scope.go:117] "RemoveContainer" containerID="6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253"
Sep 16 12:38:36.956464 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:38:36.956447    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253\": container with ID starting with 6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253 not found: ID does not exist" containerID="6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253"
Sep 16 12:38:36.956503 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.956473    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253"} err="failed to get container status \"6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253\": rpc error: code = NotFound desc = could not find container \"6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253\": container with ID starting with 6f35f35322e8be183921a64e2cc403379f1c8dd8bb72ef7aea001ba124a95253 not found: ID does not exist"
Sep 16 12:38:36.965760 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.965736    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d8575c494-dgkbx"]
Sep 16 12:38:36.967690 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:36.967671    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d8575c494-dgkbx"]
Sep 16 12:38:38.884168 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:38:38.884135    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76a0cc0b-da61-41fb-b849-ce7593a81023" path="/var/lib/kubelet/pods/76a0cc0b-da61-41fb-b849-ce7593a81023/volumes"
Sep 16 12:41:26.268544 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.268486    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"]
Sep 16 12:41:26.268948 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.268850    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a0cc0b-da61-41fb-b849-ce7593a81023" containerName="manager"
Sep 16 12:41:26.268948 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.268863    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a0cc0b-da61-41fb-b849-ce7593a81023" containerName="manager"
Sep 16 12:41:26.268948 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.268877    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy"
Sep 16 12:41:26.268948 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.268883    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="776542f7-9926-46eb-94ca-21b473d3f7bc" containerName="istio-proxy"
Sep 16 12:41:26.268948 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.268942    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a0cc0b-da61-41fb-b849-ce7593a81023" containerName="manager"
Sep 16 12:41:26.271875 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.271859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.275406 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.275385    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2f4b0fac-kserve-self-signed-certs\""
Sep 16 12:41:26.309082 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.309042    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"]
Sep 16 12:41:26.347865 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.347842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmgh\" (UniqueName: \"kubernetes.io/projected/06e6f550-8fd3-441b-9451-562b60f8cf34-kube-api-access-rsmgh\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.347964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.347870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-tmp-dir\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.347964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.347927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-home\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.347964 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.347943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-dshm\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.348068 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.347969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-model-cache\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.348103 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.348090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e6f550-8fd3-441b-9451-562b60f8cf34-tls-certs\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.448766 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.448742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-dshm\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.448876 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.448773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-model-cache\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.448876 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.448841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e6f550-8fd3-441b-9451-562b60f8cf34-tls-certs\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.448959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.448881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmgh\" (UniqueName: \"kubernetes.io/projected/06e6f550-8fd3-441b-9451-562b60f8cf34-kube-api-access-rsmgh\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.448959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.448905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-tmp-dir\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.448959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.448949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-home\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.449290 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.449266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-model-cache\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.449365 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.449299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-tmp-dir\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.449365 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.449330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-home\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.451119 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.451103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-dshm\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.451458 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.451441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e6f550-8fd3-441b-9451-562b60f8cf34-tls-certs\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.460056 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.460037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmgh\" (UniqueName: \"kubernetes.io/projected/06e6f550-8fd3-441b-9451-562b60f8cf34-kube-api-access-rsmgh\") pod \"e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") " pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.580391 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.580277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:26.867986 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:26.867962    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"]
Sep 16 12:41:26.869474 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:41:26.869440    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06e6f550_8fd3_441b_9451_562b60f8cf34.slice/crio-3da2b7c993930457ed6b77ba022927181f54fae51d62655744342697c42c9b24 WatchSource:0}: Error finding container 3da2b7c993930457ed6b77ba022927181f54fae51d62655744342697c42c9b24: Status 404 returned error can't find the container with id 3da2b7c993930457ed6b77ba022927181f54fae51d62655744342697c42c9b24
Sep 16 12:41:27.811783 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:27.811744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57" event={"ID":"06e6f550-8fd3-441b-9451-562b60f8cf34","Type":"ContainerStarted","Data":"23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19"}
Sep 16 12:41:27.812224 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:27.811792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57" event={"ID":"06e6f550-8fd3-441b-9451-562b60f8cf34","Type":"ContainerStarted","Data":"3da2b7c993930457ed6b77ba022927181f54fae51d62655744342697c42c9b24"}
Sep 16 12:41:27.848051 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:27.848000    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57" podStartSLOduration=1.8479858089999999 podStartE2EDuration="1.847985809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:27.845809873 +0000 UTC m=+2377.483134710" watchObservedRunningTime="2026-09-16 12:41:27.847985809 +0000 UTC m=+2377.485310645"
Sep 16 12:41:34.329075 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.329005    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"]
Sep 16 12:41:34.332432 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.332411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.335941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.335922    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2f4b0fac-kserve-self-signed-certs\""
Sep 16 12:41:34.358275 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.358249    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"]
Sep 16 12:41:34.416145 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.416118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-tmp-dir\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.416274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.416165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-model-cache\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.416274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.416189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4a3898-5fb6-45da-a557-487080b182a9-tls-certs\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.416274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.416225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-home\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.416274 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.416267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-dshm\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.416413 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.416291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7rp\" (UniqueName: \"kubernetes.io/projected/fc4a3898-5fb6-45da-a557-487080b182a9-kube-api-access-pv7rp\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.517578 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.517543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4a3898-5fb6-45da-a557-487080b182a9-tls-certs\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.517730 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.517592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-home\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.517730 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.517651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-dshm\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.517730 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.517690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7rp\" (UniqueName: \"kubernetes.io/projected/fc4a3898-5fb6-45da-a557-487080b182a9-kube-api-access-pv7rp\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.517900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.517728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-tmp-dir\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.517900 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.517883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-model-cache\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.518161 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.518127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-tmp-dir\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.518272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.518172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-home\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.518272 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.518254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-model-cache\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.520211 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.520185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4a3898-5fb6-45da-a557-487080b182a9-tls-certs\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.520211 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.520198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-dshm\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.528657 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.528639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7rp\" (UniqueName: \"kubernetes.io/projected/fc4a3898-5fb6-45da-a557-487080b182a9-kube-api-access-pv7rp\") pod \"e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") " pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.641720 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.641647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:34.920001 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:34.919869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"]
Sep 16 12:41:34.924657 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:41:34.924630    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc4a3898_5fb6_45da_a557_487080b182a9.slice/crio-fd1e009fe724adbf42aa5292050f76201f3656b2cd58cbdc9e1a87ff8e6e162a WatchSource:0}: Error finding container fd1e009fe724adbf42aa5292050f76201f3656b2cd58cbdc9e1a87ff8e6e162a: Status 404 returned error can't find the container with id fd1e009fe724adbf42aa5292050f76201f3656b2cd58cbdc9e1a87ff8e6e162a
Sep 16 12:41:35.852231 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:35.852187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r" event={"ID":"fc4a3898-5fb6-45da-a557-487080b182a9","Type":"ContainerStarted","Data":"52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f"}
Sep 16 12:41:35.852231 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:35.852232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r" event={"ID":"fc4a3898-5fb6-45da-a557-487080b182a9","Type":"ContainerStarted","Data":"fd1e009fe724adbf42aa5292050f76201f3656b2cd58cbdc9e1a87ff8e6e162a"}
Sep 16 12:41:35.893398 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:35.893349    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r" podStartSLOduration=1.89333364 podStartE2EDuration="1.89333364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:35.891667557 +0000 UTC m=+2385.528992396" watchObservedRunningTime="2026-09-16 12:41:35.89333364 +0000 UTC m=+2385.530658478"
Sep 16 12:41:36.580825 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:36.580785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:36.580825 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:36.580830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:36.593018 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:36.592986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:36.867851 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:36.867756    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:41:44.642491 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:44.642453    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:44.642932 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:44.642504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:44.654895 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:44.654865    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:44.905931 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:44.905843    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:41:50.576372 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:50.576337    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"]
Sep 16 12:41:51.071840 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:51.071811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:41:51.081381 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:51.081358    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:41:52.615729 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:41:52.615694    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"]
Sep 16 12:42:05.595261 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.595222    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57" podUID="06e6f550-8fd3-441b-9451-562b60f8cf34" containerName="main" containerID="cri-o://23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19" gracePeriod=45
Sep 16 12:42:05.844408 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.844385    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:42:05.896667 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.896588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e6f550-8fd3-441b-9451-562b60f8cf34-tls-certs\") pod \"06e6f550-8fd3-441b-9451-562b60f8cf34\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") "
Sep 16 12:42:05.896667 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.896639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-model-cache\") pod \"06e6f550-8fd3-441b-9451-562b60f8cf34\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") "
Sep 16 12:42:05.896868 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.896670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-home\") pod \"06e6f550-8fd3-441b-9451-562b60f8cf34\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") "
Sep 16 12:42:05.896868 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.896724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-tmp-dir\") pod \"06e6f550-8fd3-441b-9451-562b60f8cf34\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") "
Sep 16 12:42:05.896868 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.896759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-dshm\") pod \"06e6f550-8fd3-441b-9451-562b60f8cf34\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") "
Sep 16 12:42:05.896868 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.896789    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsmgh\" (UniqueName: \"kubernetes.io/projected/06e6f550-8fd3-441b-9451-562b60f8cf34-kube-api-access-rsmgh\") pod \"06e6f550-8fd3-441b-9451-562b60f8cf34\" (UID: \"06e6f550-8fd3-441b-9451-562b60f8cf34\") "
Sep 16 12:42:05.897093 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.896889    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-model-cache" (OuterVolumeSpecName: "model-cache") pod "06e6f550-8fd3-441b-9451-562b60f8cf34" (UID: "06e6f550-8fd3-441b-9451-562b60f8cf34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:05.897093 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.896971    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06e6f550-8fd3-441b-9451-562b60f8cf34" (UID: "06e6f550-8fd3-441b-9451-562b60f8cf34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:05.897188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.897167    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:05.897236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.897189    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:05.897408 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.897379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-home" (OuterVolumeSpecName: "home") pod "06e6f550-8fd3-441b-9451-562b60f8cf34" (UID: "06e6f550-8fd3-441b-9451-562b60f8cf34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:05.899748 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.899715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e6f550-8fd3-441b-9451-562b60f8cf34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06e6f550-8fd3-441b-9451-562b60f8cf34" (UID: "06e6f550-8fd3-441b-9451-562b60f8cf34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:05.899748 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.899737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e6f550-8fd3-441b-9451-562b60f8cf34-kube-api-access-rsmgh" (OuterVolumeSpecName: "kube-api-access-rsmgh") pod "06e6f550-8fd3-441b-9451-562b60f8cf34" (UID: "06e6f550-8fd3-441b-9451-562b60f8cf34"). InnerVolumeSpecName "kube-api-access-rsmgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:42:05.899928 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.899768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-dshm" (OuterVolumeSpecName: "dshm") pod "06e6f550-8fd3-441b-9451-562b60f8cf34" (UID: "06e6f550-8fd3-441b-9451-562b60f8cf34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:05.997619 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.997596    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e6f550-8fd3-441b-9451-562b60f8cf34-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:05.997619 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.997617    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:05.997619 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.997625    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e6f550-8fd3-441b-9451-562b60f8cf34-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:05.997875 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.997637    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsmgh\" (UniqueName: \"kubernetes.io/projected/06e6f550-8fd3-441b-9451-562b60f8cf34-kube-api-access-rsmgh\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:05.998126 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.998100    2578 generic.go:356] "Generic (PLEG): container finished" podID="06e6f550-8fd3-441b-9451-562b60f8cf34" containerID="23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19" exitCode=0
Sep 16 12:42:05.998220 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.998160    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"
Sep 16 12:42:05.998220 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.998178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57" event={"ID":"06e6f550-8fd3-441b-9451-562b60f8cf34","Type":"ContainerDied","Data":"23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19"}
Sep 16 12:42:05.998220 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.998218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57" event={"ID":"06e6f550-8fd3-441b-9451-562b60f8cf34","Type":"ContainerDied","Data":"3da2b7c993930457ed6b77ba022927181f54fae51d62655744342697c42c9b24"}
Sep 16 12:42:05.998342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:05.998236    2578 scope.go:117] "RemoveContainer" containerID="23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19"
Sep 16 12:42:06.010874 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:06.010856    2578 scope.go:117] "RemoveContainer" containerID="23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19"
Sep 16 12:42:06.011149 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:42:06.011132    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19\": container with ID starting with 23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19 not found: ID does not exist" containerID="23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19"
Sep 16 12:42:06.011189 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:06.011157    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19"} err="failed to get container status \"23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19\": rpc error: code = NotFound desc = could not find container \"23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19\": container with ID starting with 23ab160367e4a6ea4c6d8d8b9096dc028791dfbe22d3738d621d05ae05506b19 not found: ID does not exist"
Sep 16 12:42:06.027623 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:06.027602    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"]
Sep 16 12:42:06.032916 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:06.032896    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2f4b0fac-kserve-68c77b75ff-s9v57"]
Sep 16 12:42:06.883811 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:06.883781    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06e6f550-8fd3-441b-9451-562b60f8cf34" path="/var/lib/kubelet/pods/06e6f550-8fd3-441b-9451-562b60f8cf34/volumes"
Sep 16 12:42:07.634367 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:07.634325    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r" podUID="fc4a3898-5fb6-45da-a557-487080b182a9" containerName="main" containerID="cri-o://52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f" gracePeriod=45
Sep 16 12:42:07.899942 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:07.899915    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:42:08.011055 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.011021    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc4a3898-5fb6-45da-a557-487080b182a9" containerID="52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f" exitCode=0
Sep 16 12:42:08.011248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.011064    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"
Sep 16 12:42:08.011248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.011110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r" event={"ID":"fc4a3898-5fb6-45da-a557-487080b182a9","Type":"ContainerDied","Data":"52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f"}
Sep 16 12:42:08.011248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.011156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r" event={"ID":"fc4a3898-5fb6-45da-a557-487080b182a9","Type":"ContainerDied","Data":"fd1e009fe724adbf42aa5292050f76201f3656b2cd58cbdc9e1a87ff8e6e162a"}
Sep 16 12:42:08.011248 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.011179    2578 scope.go:117] "RemoveContainer" containerID="52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f"
Sep 16 12:42:08.015061 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015036    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-tmp-dir\") pod \"fc4a3898-5fb6-45da-a557-487080b182a9\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") "
Sep 16 12:42:08.015169 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-dshm\") pod \"fc4a3898-5fb6-45da-a557-487080b182a9\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") "
Sep 16 12:42:08.015169 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-model-cache\") pod \"fc4a3898-5fb6-45da-a557-487080b182a9\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") "
Sep 16 12:42:08.015169 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4a3898-5fb6-45da-a557-487080b182a9-tls-certs\") pod \"fc4a3898-5fb6-45da-a557-487080b182a9\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") "
Sep 16 12:42:08.015342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015222    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-home\") pod \"fc4a3898-5fb6-45da-a557-487080b182a9\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") "
Sep 16 12:42:08.015342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv7rp\" (UniqueName: \"kubernetes.io/projected/fc4a3898-5fb6-45da-a557-487080b182a9-kube-api-access-pv7rp\") pod \"fc4a3898-5fb6-45da-a557-487080b182a9\" (UID: \"fc4a3898-5fb6-45da-a557-487080b182a9\") "
Sep 16 12:42:08.015342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc4a3898-5fb6-45da-a557-487080b182a9" (UID: "fc4a3898-5fb6-45da-a557-487080b182a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:08.015488 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc4a3898-5fb6-45da-a557-487080b182a9" (UID: "fc4a3898-5fb6-45da-a557-487080b182a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:08.015587 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015566    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-tmp-dir\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:08.015646 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015588    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-model-cache\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:08.015646 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.015582    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-home" (OuterVolumeSpecName: "home") pod "fc4a3898-5fb6-45da-a557-487080b182a9" (UID: "fc4a3898-5fb6-45da-a557-487080b182a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:08.017460 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.017435    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-dshm" (OuterVolumeSpecName: "dshm") pod "fc4a3898-5fb6-45da-a557-487080b182a9" (UID: "fc4a3898-5fb6-45da-a557-487080b182a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:08.017631 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.017541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4a3898-5fb6-45da-a557-487080b182a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc4a3898-5fb6-45da-a557-487080b182a9" (UID: "fc4a3898-5fb6-45da-a557-487080b182a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:08.017698 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.017645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4a3898-5fb6-45da-a557-487080b182a9-kube-api-access-pv7rp" (OuterVolumeSpecName: "kube-api-access-pv7rp") pod "fc4a3898-5fb6-45da-a557-487080b182a9" (UID: "fc4a3898-5fb6-45da-a557-487080b182a9"). InnerVolumeSpecName "kube-api-access-pv7rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:42:08.020931 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.020898    2578 scope.go:117] "RemoveContainer" containerID="52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f"
Sep 16 12:42:08.021243 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:42:08.021221    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f\": container with ID starting with 52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f not found: ID does not exist" containerID="52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f"
Sep 16 12:42:08.021315 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.021252    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f"} err="failed to get container status \"52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f\": rpc error: code = NotFound desc = could not find container \"52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f\": container with ID starting with 52684a9549c8c3a0de752af46f30bfe06222cbfbaf05c3bccb41d65d288c465f not found: ID does not exist"
Sep 16 12:42:08.116342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.116318    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-dshm\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:08.116342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.116338    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4a3898-5fb6-45da-a557-487080b182a9-tls-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:08.116342 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.116346    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc4a3898-5fb6-45da-a557-487080b182a9-home\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:08.116548 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.116355    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv7rp\" (UniqueName: \"kubernetes.io/projected/fc4a3898-5fb6-45da-a557-487080b182a9-kube-api-access-pv7rp\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:08.338008 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.337978    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"]
Sep 16 12:42:08.345362 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.345338    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2f4b0fac-kserve-cd9c49c9f-x8t7r"]
Sep 16 12:42:08.884074 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:08.884042    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4a3898-5fb6-45da-a557-487080b182a9" path="/var/lib/kubelet/pods/fc4a3898-5fb6-45da-a557-487080b182a9/volumes"
Sep 16 12:42:16.396312 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.396235    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"]
Sep 16 12:42:16.396714 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.396623    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e6f550-8fd3-441b-9451-562b60f8cf34" containerName="main"
Sep 16 12:42:16.396714 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.396637    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e6f550-8fd3-441b-9451-562b60f8cf34" containerName="main"
Sep 16 12:42:16.396714 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.396648    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4a3898-5fb6-45da-a557-487080b182a9" containerName="main"
Sep 16 12:42:16.396714 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.396654    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4a3898-5fb6-45da-a557-487080b182a9" containerName="main"
Sep 16 12:42:16.396855 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.396720    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06e6f550-8fd3-441b-9451-562b60f8cf34" containerName="main"
Sep 16 12:42:16.396855 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.396728    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4a3898-5fb6-45da-a557-487080b182a9" containerName="main"
Sep 16 12:42:16.401129 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.401110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.427749 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.427720    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"]
Sep 16 12:42:16.485996 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.485964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-maas-api-tls\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.486135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.486015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-openshift-service-ca\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.486135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.486063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-metrics-certs\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.486135 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.486107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5mf\" (UniqueName: \"kubernetes.io/projected/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-kube-api-access-fz5mf\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.586541 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.586497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-openshift-service-ca\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.586541 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.586541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-metrics-certs\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.586735 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.586561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5mf\" (UniqueName: \"kubernetes.io/projected/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-kube-api-access-fz5mf\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.586735 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.586623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-maas-api-tls\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.587304 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.587284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-openshift-service-ca\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.589073 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.589034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-metrics-certs\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.589187 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.589111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-maas-api-tls\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.601350 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.601315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5mf\" (UniqueName: \"kubernetes.io/projected/4ce87d5e-db4d-4ab0-ac4a-12770f83322f-kube-api-access-fz5mf\") pod \"maas-api-88dfc4b4f-chmnc\" (UID: \"4ce87d5e-db4d-4ab0-ac4a-12770f83322f\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.710941 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.710915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:16.995357 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:16.995331    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"]
Sep 16 12:42:16.997269 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:42:16.997245    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce87d5e_db4d_4ab0_ac4a_12770f83322f.slice/crio-6d6ef324f2d0d927bb933a99923e178c0585aa1a76456be83e858665d03ba42b WatchSource:0}: Error finding container 6d6ef324f2d0d927bb933a99923e178c0585aa1a76456be83e858665d03ba42b: Status 404 returned error can't find the container with id 6d6ef324f2d0d927bb933a99923e178c0585aa1a76456be83e858665d03ba42b
Sep 16 12:42:17.058128 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:17.058094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc" event={"ID":"4ce87d5e-db4d-4ab0-ac4a-12770f83322f","Type":"ContainerStarted","Data":"6d6ef324f2d0d927bb933a99923e178c0585aa1a76456be83e858665d03ba42b"}
Sep 16 12:42:18.065200 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:18.065161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc" event={"ID":"4ce87d5e-db4d-4ab0-ac4a-12770f83322f","Type":"ContainerStarted","Data":"59d00d5abad7ee45a39f7f4ad800073d0da5dfc3646899c8e0e6b11d4fc8ce25"}
Sep 16 12:42:18.065696 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:18.065320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:18.099888 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:18.099833    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc" podStartSLOduration=1.6115681130000001 podStartE2EDuration="2.09981975s" totalImagesPullingTime="488.251637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:42:18.097770514 +0000 UTC m=+2427.735095351" watchObservedRunningTime="2026-09-16 12:42:18.09981975 +0000 UTC m=+2427.737144588"
Sep 16 12:42:24.075053 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:24.075022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-chmnc"
Sep 16 12:42:24.171139 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:24.171108    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"]
Sep 16 12:42:24.171365 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:24.171321    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5" podUID="d5788b24-c547-42f5-8774-c5c8306065c5" containerName="maas-api" containerID="cri-o://dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec" gracePeriod=30
Sep 16 12:42:24.660881 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:24.660836    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5" podUID="d5788b24-c547-42f5-8774-c5c8306065c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.44:8443/health\": dial tcp 10.132.0.44:8443: connect: connection refused"
Sep 16 12:42:25.420855 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.420830    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:42:25.560504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.560415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5788b24-c547-42f5-8774-c5c8306065c5-openshift-service-ca\") pod \"d5788b24-c547-42f5-8774-c5c8306065c5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") "
Sep 16 12:42:25.560504 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.560494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-metrics-certs\") pod \"d5788b24-c547-42f5-8774-c5c8306065c5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") "
Sep 16 12:42:25.560752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.560608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgmm6\" (UniqueName: \"kubernetes.io/projected/d5788b24-c547-42f5-8774-c5c8306065c5-kube-api-access-xgmm6\") pod \"d5788b24-c547-42f5-8774-c5c8306065c5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") "
Sep 16 12:42:25.560752 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.560649    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-maas-api-tls\") pod \"d5788b24-c547-42f5-8774-c5c8306065c5\" (UID: \"d5788b24-c547-42f5-8774-c5c8306065c5\") "
Sep 16 12:42:25.560846 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.560807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5788b24-c547-42f5-8774-c5c8306065c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5788b24-c547-42f5-8774-c5c8306065c5" (UID: "d5788b24-c547-42f5-8774-c5c8306065c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:25.560959 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.560941    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5788b24-c547-42f5-8774-c5c8306065c5-openshift-service-ca\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:25.562697 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.562667    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5788b24-c547-42f5-8774-c5c8306065c5" (UID: "d5788b24-c547-42f5-8774-c5c8306065c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:25.562836 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.562819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5788b24-c547-42f5-8774-c5c8306065c5" (UID: "d5788b24-c547-42f5-8774-c5c8306065c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:25.562881 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.562837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5788b24-c547-42f5-8774-c5c8306065c5-kube-api-access-xgmm6" (OuterVolumeSpecName: "kube-api-access-xgmm6") pod "d5788b24-c547-42f5-8774-c5c8306065c5" (UID: "d5788b24-c547-42f5-8774-c5c8306065c5"). InnerVolumeSpecName "kube-api-access-xgmm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:42:25.661366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.661335    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-metrics-certs\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:25.661366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.661361    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmm6\" (UniqueName: \"kubernetes.io/projected/d5788b24-c547-42f5-8774-c5c8306065c5-kube-api-access-xgmm6\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:25.661366 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:25.661371    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5788b24-c547-42f5-8774-c5c8306065c5-maas-api-tls\") on node \"ip-10-0-136-170.ec2.internal\" DevicePath \"\""
Sep 16 12:42:26.105918 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.105883    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5788b24-c547-42f5-8774-c5c8306065c5" containerID="dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec" exitCode=0
Sep 16 12:42:26.106064 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.105927    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"
Sep 16 12:42:26.106064 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.105972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5" event={"ID":"d5788b24-c547-42f5-8774-c5c8306065c5","Type":"ContainerDied","Data":"dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec"}
Sep 16 12:42:26.106064 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.106024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5" event={"ID":"d5788b24-c547-42f5-8774-c5c8306065c5","Type":"ContainerDied","Data":"05a130c3ea9553df4c7e74f4bb8761b406b1964cdffef08bc31198fe0eb68b3c"}
Sep 16 12:42:26.106064 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.106048    2578 scope.go:117] "RemoveContainer" containerID="dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec"
Sep 16 12:42:26.118611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.118592    2578 scope.go:117] "RemoveContainer" containerID="dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec"
Sep 16 12:42:26.118912 ip-10-0-136-170 kubenswrapper[2578]: E0916 12:42:26.118890    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec\": container with ID starting with dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec not found: ID does not exist" containerID="dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec"
Sep 16 12:42:26.118999 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.118918    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec"} err="failed to get container status \"dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec\": rpc error: code = NotFound desc = could not find container \"dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec\": container with ID starting with dff707e547263a3d9b1fed86e4a555686b8c78c5f565f4a77fb843c70dbac1ec not found: ID does not exist"
Sep 16 12:42:26.137403 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.137379    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"]
Sep 16 12:42:26.144611 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.144589    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-jbdg5"]
Sep 16 12:42:26.883736 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:26.883705    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5788b24-c547-42f5-8774-c5c8306065c5" path="/var/lib/kubelet/pods/d5788b24-c547-42f5-8774-c5c8306065c5/volumes"
Sep 16 12:42:46.569492 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:46.569463    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74b9d88877-srlcv_22a51a5f-252b-4927-bd48-67ac0c1cd524/authorino/0.log"
Sep 16 12:42:54.223939 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:54.223911    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b54c497cf-pdcp9_a5ffc7b4-3b52-402a-a813-d8f8c737f09d/copy-manifests/0.log"
Sep 16 12:42:54.232992 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:54.232965    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b54c497cf-pdcp9_a5ffc7b4-3b52-402a-a813-d8f8c737f09d/manager/0.log"
Sep 16 12:42:54.777571 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:54.777508    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-88dfc4b4f-chmnc_4ce87d5e-db4d-4ab0-ac4a-12770f83322f/maas-api/0.log"
Sep 16 12:42:56.207081 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:56.207050    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74b9d88877-srlcv_22a51a5f-252b-4927-bd48-67ac0c1cd524/authorino/0.log"
Sep 16 12:42:56.681557 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:56.681466    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xrpbn_8392efaa-bdc8-44a0-9fd9-d264ecb8de3d/kuadrant-console-plugin/0.log"
Sep 16 12:42:57.894371 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:57.894328    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d74dbbc-hbzgj_57927c1d-b69f-42e5-943f-41a163bdd5a2/kube-auth-proxy/0.log"
Sep 16 12:42:58.580742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:58.580714    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-544f785cb6-6wc5m_de22da40-9f25-4233-87ff-9fbc536d1a7e/router/0.log"
Sep 16 12:42:58.989464 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:58.989434    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9hhht_f768c5bc-d430-487d-8052-7920af9aa205/main/0.log"
Sep 16 12:42:59.247334 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:59.247244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-vgxt8_d25310be-d0cc-412a-881f-657aef923a53/main/0.log"
Sep 16 12:42:59.753560 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:42:59.753532    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-rlcqw_e8b212f8-e066-4ea3-b033-ea1e503e8188/main/0.log"
Sep 16 12:43:06.162680 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.162646    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8nlh7/must-gather-pxktz"]
Sep 16 12:43:06.163063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.162955    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5788b24-c547-42f5-8774-c5c8306065c5" containerName="maas-api"
Sep 16 12:43:06.163063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.162965    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5788b24-c547-42f5-8774-c5c8306065c5" containerName="maas-api"
Sep 16 12:43:06.163063 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.163050    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5788b24-c547-42f5-8774-c5c8306065c5" containerName="maas-api"
Sep 16 12:43:06.166129 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.166112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8nlh7/must-gather-pxktz"
Sep 16 12:43:06.169410 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.169383    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8nlh7\"/\"openshift-service-ca.crt\""
Sep 16 12:43:06.169410 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.169383    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8nlh7\"/\"default-dockercfg-x94cz\""
Sep 16 12:43:06.171236 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.171216    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8nlh7\"/\"kube-root-ca.crt\""
Sep 16 12:43:06.183642 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.183622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69m2m\" (UniqueName: \"kubernetes.io/projected/bb798742-f62d-40c4-806b-3f4361e58bfc-kube-api-access-69m2m\") pod \"must-gather-pxktz\" (UID: \"bb798742-f62d-40c4-806b-3f4361e58bfc\") " pod="openshift-must-gather-8nlh7/must-gather-pxktz"
Sep 16 12:43:06.183756 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.183656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb798742-f62d-40c4-806b-3f4361e58bfc-must-gather-output\") pod \"must-gather-pxktz\" (UID: \"bb798742-f62d-40c4-806b-3f4361e58bfc\") " pod="openshift-must-gather-8nlh7/must-gather-pxktz"
Sep 16 12:43:06.218686 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.218664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8nlh7/must-gather-pxktz"]
Sep 16 12:43:06.284904 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.284872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69m2m\" (UniqueName: \"kubernetes.io/projected/bb798742-f62d-40c4-806b-3f4361e58bfc-kube-api-access-69m2m\") pod \"must-gather-pxktz\" (UID: \"bb798742-f62d-40c4-806b-3f4361e58bfc\") " pod="openshift-must-gather-8nlh7/must-gather-pxktz"
Sep 16 12:43:06.284904 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.284905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb798742-f62d-40c4-806b-3f4361e58bfc-must-gather-output\") pod \"must-gather-pxktz\" (UID: \"bb798742-f62d-40c4-806b-3f4361e58bfc\") " pod="openshift-must-gather-8nlh7/must-gather-pxktz"
Sep 16 12:43:06.285252 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.285236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb798742-f62d-40c4-806b-3f4361e58bfc-must-gather-output\") pod \"must-gather-pxktz\" (UID: \"bb798742-f62d-40c4-806b-3f4361e58bfc\") " pod="openshift-must-gather-8nlh7/must-gather-pxktz"
Sep 16 12:43:06.305883 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.305856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69m2m\" (UniqueName: \"kubernetes.io/projected/bb798742-f62d-40c4-806b-3f4361e58bfc-kube-api-access-69m2m\") pod \"must-gather-pxktz\" (UID: \"bb798742-f62d-40c4-806b-3f4361e58bfc\") " pod="openshift-must-gather-8nlh7/must-gather-pxktz"
Sep 16 12:43:06.474421 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.474393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8nlh7/must-gather-pxktz"
Sep 16 12:43:06.759163 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:06.759130    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8nlh7/must-gather-pxktz"]
Sep 16 12:43:06.760013 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:43:06.759983    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb798742_f62d_40c4_806b_3f4361e58bfc.slice/crio-bd3e904e719db776ea9858d5d47e2d39570997c9ff4a3beb4ebe32393bbbe31a WatchSource:0}: Error finding container bd3e904e719db776ea9858d5d47e2d39570997c9ff4a3beb4ebe32393bbbe31a: Status 404 returned error can't find the container with id bd3e904e719db776ea9858d5d47e2d39570997c9ff4a3beb4ebe32393bbbe31a
Sep 16 12:43:07.316352 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:07.316316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8nlh7/must-gather-pxktz" event={"ID":"bb798742-f62d-40c4-806b-3f4361e58bfc","Type":"ContainerStarted","Data":"bd3e904e719db776ea9858d5d47e2d39570997c9ff4a3beb4ebe32393bbbe31a"}
Sep 16 12:43:08.323742 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:08.323710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8nlh7/must-gather-pxktz" event={"ID":"bb798742-f62d-40c4-806b-3f4361e58bfc","Type":"ContainerStarted","Data":"c652dfaca7ecbafb86f945221888502cb852b2f220f94d32ab20e4f4147f57d2"}
Sep 16 12:43:08.324188 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:08.323757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8nlh7/must-gather-pxktz" event={"ID":"bb798742-f62d-40c4-806b-3f4361e58bfc","Type":"ContainerStarted","Data":"acd62ec4c7364f539a554c3f62ca39f6e6abdb77d491f5477e2ebfe18f3eb195"}
Sep 16 12:43:08.389222 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:08.389165    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8nlh7/must-gather-pxktz" podStartSLOduration=1.564173287 podStartE2EDuration="2.389149458s" totalImagesPullingTime="824.976171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:43:08.384907686 +0000 UTC m=+2478.022232524" watchObservedRunningTime="2026-09-16 12:43:08.389149458 +0000 UTC m=+2478.026474296"
Sep 16 12:43:09.810073 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:09.810034    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8fst2_04c58447-61b8-400e-be4b-8519523f5788/global-pull-secret-syncer/0.log"
Sep 16 12:43:10.059307 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:10.059269    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hcxdk_0dfca71f-6cb4-432e-9a94-339724df3d63/konnectivity-agent/0.log"
Sep 16 12:43:10.157483 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:10.157380    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-170.ec2.internal_e474ffa0780843092bf4057d45da8093/haproxy/0.log"
Sep 16 12:43:15.316932 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:15.316898    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74b9d88877-srlcv_22a51a5f-252b-4927-bd48-67ac0c1cd524/authorino/0.log"
Sep 16 12:43:15.464332 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:15.464286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xrpbn_8392efaa-bdc8-44a0-9fd9-d264ecb8de3d/kuadrant-console-plugin/0.log"
Sep 16 12:43:18.135796 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:18.135763    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-rx5ch_5ad945a4-0cf4-4e6b-99e8-8a21ce8323c9/cluster-monitoring-operator/0.log"
Sep 16 12:43:18.453269 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:18.453186    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8tz2_4fd22cc8-bcd2-489e-b1e4-591b8078ca14/node-exporter/0.log"
Sep 16 12:43:18.481654 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:18.481626    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8tz2_4fd22cc8-bcd2-489e-b1e4-591b8078ca14/kube-rbac-proxy/0.log"
Sep 16 12:43:18.535829 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:18.535801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8tz2_4fd22cc8-bcd2-489e-b1e4-591b8078ca14/init-textfile/0.log"
Sep 16 12:43:18.924429 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:18.924383    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"]
Sep 16 12:43:18.928048 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:18.928030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:18.964585 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:18.964546    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"]
Sep 16 12:43:19.030419 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.030384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjtxh\" (UniqueName: \"kubernetes.io/projected/5c418484-9710-4589-8a07-56613fcc3240-kube-api-access-pjtxh\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.030419 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.030419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-podres\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.030705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.030561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-proc\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.030705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.030589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-lib-modules\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.030705 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.030621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-sys\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.131933 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.131895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-podres\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.132148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.131971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-proc\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.132148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.131994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-lib-modules\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.132148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.132022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-sys\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.132148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.132048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-proc\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.132148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.132080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjtxh\" (UniqueName: \"kubernetes.io/projected/5c418484-9710-4589-8a07-56613fcc3240-kube-api-access-pjtxh\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.132148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.132088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-podres\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.132148 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.132115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-sys\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.132573 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.132158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c418484-9710-4589-8a07-56613fcc3240-lib-modules\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.146575 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.146470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjtxh\" (UniqueName: \"kubernetes.io/projected/5c418484-9710-4589-8a07-56613fcc3240-kube-api-access-pjtxh\") pod \"perf-node-gather-daemonset-8b66t\" (UID: \"5c418484-9710-4589-8a07-56613fcc3240\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.238606 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.238563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:19.304420 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.304378    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-hjqqz_9fefc4a2-e8f1-4a3d-b114-78c1cc1d6779/prometheus-operator-admission-webhook/0.log"
Sep 16 12:43:19.535073 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:19.534971    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"]
Sep 16 12:43:19.541496 ip-10-0-136-170 kubenswrapper[2578]: W0916 12:43:19.541460    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c418484_9710_4589_8a07_56613fcc3240.slice/crio-fa5f52eb6fe0eef37c248b6457e8a5b7e9f95c535aca8a6f7d759685b3878c24 WatchSource:0}: Error finding container fa5f52eb6fe0eef37c248b6457e8a5b7e9f95c535aca8a6f7d759685b3878c24: Status 404 returned error can't find the container with id fa5f52eb6fe0eef37c248b6457e8a5b7e9f95c535aca8a6f7d759685b3878c24
Sep 16 12:43:20.424300 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:20.424266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t" event={"ID":"5c418484-9710-4589-8a07-56613fcc3240","Type":"ContainerStarted","Data":"44df58f8f9db7b59acf561820bbb66f16d70de6c0d64b4c6877ee3d0568f94b6"}
Sep 16 12:43:20.424940 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:20.424915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t" event={"ID":"5c418484-9710-4589-8a07-56613fcc3240","Type":"ContainerStarted","Data":"fa5f52eb6fe0eef37c248b6457e8a5b7e9f95c535aca8a6f7d759685b3878c24"}
Sep 16 12:43:20.425763 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:20.425727    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:20.448082 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:20.448030    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t" podStartSLOduration=2.448012429 podStartE2EDuration="2.448012429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:43:20.445657522 +0000 UTC m=+2490.082982362" watchObservedRunningTime="2026-09-16 12:43:20.448012429 +0000 UTC m=+2490.085337268"
Sep 16 12:43:21.463228 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:21.463201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/2.log"
Sep 16 12:43:21.464046 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:21.463534    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8ls9j_d2bd3fc1-6334-453c-8ca1-048c8b77bc17/console-operator/1.log"
Sep 16 12:43:21.995346 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:21.995309    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-t8qqw_a35c46ce-8537-451f-97c6-f31e09906543/download-server/0.log"
Sep 16 12:43:22.610255 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:22.610228    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-pkctg_3ef15515-9c47-49b7-bd0e-9832d387fee7/volume-data-source-validator/0.log"
Sep 16 12:43:23.677629 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:23.677598    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qz5zw_acd2a478-2880-4d82-8a20-70e1e6a538c0/dns/0.log"
Sep 16 12:43:23.701126 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:23.701100    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qz5zw_acd2a478-2880-4d82-8a20-70e1e6a538c0/kube-rbac-proxy/0.log"
Sep 16 12:43:23.872687 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:23.872657    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kw862_4dbddbd3-dda8-46a7-b3ac-a4d8fada3f1f/dns-node-resolver/0.log"
Sep 16 12:43:24.632471 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:24.632444    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wt7g7_ef67b5a6-8ba4-41b0-9f63-3a242a942d3c/node-ca/0.log"
Sep 16 12:43:26.076285 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:26.076255    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d74dbbc-hbzgj_57927c1d-b69f-42e5-943f-41a163bdd5a2/kube-auth-proxy/0.log"
Sep 16 12:43:26.212363 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:26.212327    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-544f785cb6-6wc5m_de22da40-9f25-4233-87ff-9fbc536d1a7e/router/0.log"
Sep 16 12:43:26.441700 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:26.441618    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-8b66t"
Sep 16 12:43:27.193416 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:27.193375    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qv7vb_9b44461a-fdcb-4a38-83d3-e72418354565/serve-healthcheck-canary/0.log"
Sep 16 12:43:27.751299 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:27.751258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-v7sp6_14f56708-c1b2-4ac8-8a6b-3eba48cfb62c/insights-operator/0.log"
Sep 16 12:43:27.753391 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:27.753362    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-v7sp6_14f56708-c1b2-4ac8-8a6b-3eba48cfb62c/insights-operator/1.log"
Sep 16 12:43:27.861105 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:27.861022    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8q8l_69ed17f4-800e-4a0b-b25d-d5e40307ee7a/kube-rbac-proxy/0.log"
Sep 16 12:43:27.893929 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:27.893897    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8q8l_69ed17f4-800e-4a0b-b25d-d5e40307ee7a/exporter/0.log"
Sep 16 12:43:27.941376 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:27.941344    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8q8l_69ed17f4-800e-4a0b-b25d-d5e40307ee7a/extractor/0.log"
Sep 16 12:43:31.296438 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:31.296398    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b54c497cf-pdcp9_a5ffc7b4-3b52-402a-a813-d8f8c737f09d/manager/0.log"
Sep 16 12:43:31.363064 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:31.362993    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b54c497cf-pdcp9_a5ffc7b4-3b52-402a-a813-d8f8c737f09d/copy-manifests/0.log"
Sep 16 12:43:31.366815 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:31.366793    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d74dbbc-hbzgj_57927c1d-b69f-42e5-943f-41a163bdd5a2/kube-auth-proxy/0.log"
Sep 16 12:43:31.476243 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:31.476213    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-544f785cb6-6wc5m_de22da40-9f25-4233-87ff-9fbc536d1a7e/router/0.log"
Sep 16 12:43:32.733968 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:32.733935    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-596fbdbcd6-4pxnd_82b4f555-e233-4ada-92d7-b3e6fa5119d0/manager/0.log"
Sep 16 12:43:33.553243 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:33.553196    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d74dbbc-hbzgj_57927c1d-b69f-42e5-943f-41a163bdd5a2/kube-auth-proxy/0.log"
Sep 16 12:43:33.664862 ip-10-0-136-170 kubenswrapper[2578]: I0916 12:43:33.664830    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-544f785cb6-6wc5m_de22da40-9f25-4233-87ff-9fbc536d1a7e/router/0.log"