Sep 10 14:27:52.036857 ip-10-0-129-162 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 14:27:52.425387 ip-10-0-129-162 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:27:52.425387 ip-10-0-129-162 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 14:27:52.425387 ip-10-0-129-162 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:27:52.425387 ip-10-0-129-162 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 14:27:52.425387 ip-10-0-129-162 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:27:52.427444 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.427346    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 14:27:52.432071 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432052    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:27:52.432071 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432067    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:27:52.432071 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432072    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:27:52.432071 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432076    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432080    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432084    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432089    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432093    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432097    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432100    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432104    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432107    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432111    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432115    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432119    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432123    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432127    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432138    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432142    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432146    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432150    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432154    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:27:52.432312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432157    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432161    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432165    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432169    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432173    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432177    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432181    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432184    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432188    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432192    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432200    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432207    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432215    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432220    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432225    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432230    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432237    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432243    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432248    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432252    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:27:52.433085 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432257    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432261    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432265    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432270    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432275    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432280    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432283    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432288    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432292    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432296    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432300    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432304    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432308    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432312    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432316    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432320    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432324    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432328    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432332    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432336    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:27:52.433939 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432340    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432344    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432349    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432352    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432357    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432361    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432365    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432370    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432374    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432380    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432384    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432389    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432393    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432397    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432401    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432405    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432409    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432416    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432421    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:27:52.434777 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432426    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432431    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432435    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432440    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432444    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432448    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432453    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432457    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432461    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432465    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432469    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432473    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432477    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432481    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432485    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432488    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432492    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.432496    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433639    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433649    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:27:52.435583 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433655    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433660    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433664    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433669    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433674    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433678    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433682    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433687    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433691    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433695    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433700    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433704    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433708    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433712    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433717    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433721    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433724    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433728    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433733    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433737    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:27:52.436430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433741    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433745    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433749    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433753    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433757    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433761    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433765    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433768    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433772    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433777    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433781    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433785    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433790    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433794    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433798    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433802    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433806    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433810    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433814    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:27:52.437101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433818    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433822    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433826    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433830    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433836    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433840    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433844    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433848    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433852    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433856    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433864    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433870    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433874    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433879    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433884    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433889    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433893    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433899    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433905    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:27:52.437593 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433910    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433914    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433919    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433923    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433928    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433932    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433936    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433940    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433945    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433949    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433953    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433960    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433966    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433971    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433975    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433979    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433983    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433987    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433991    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433995    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:27:52.438142 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.433999    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434003    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434007    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434011    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434015    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434019    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434023    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434028    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434033    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434037    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434042    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434046    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434050    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434054    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434058    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434062    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434066    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434070    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434074    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434706    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:27:52.438752 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434715    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434719    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434723    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434728    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434732    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434736    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434741    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434745    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434750    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434754    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434758    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434762    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434766    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434770    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434774    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434778    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434782    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434786    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434790    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434797    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:27:52.439251 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434801    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434805    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434813    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434817    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434821    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434825    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434829    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434833    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434837    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434841    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434845    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434849    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434853    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434857    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434865    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434871    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434877    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434881    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434886    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434890    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:27:52.439922 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434895    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434899    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434903    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434907    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434911    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434915    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434921    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434925    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434929    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434933    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434937    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434943    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434947    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434951    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434957    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434961    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434965    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434969    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434973    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434977    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:27:52.440406 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434981    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434985    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434990    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.434995    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435000    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435004    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435008    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435012    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435015    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435020    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435024    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435029    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435034    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435038    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435045    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435050    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435054    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435059    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435063    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:27:52.440937 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435067    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435071    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435075    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435079    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435084    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435088    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435092    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435098    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435102    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435106    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435111    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435115    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435119    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435123    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435127    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435131    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435135    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435139    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.435143    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435246    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435291    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 14:27:52.441445 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435300    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435307    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435315    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435320    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435327    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435334    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435339    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435344    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435349    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435354    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435359    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435364    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435368    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435372    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435377    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435384    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435391    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435396    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435402    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435407    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435413    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435419    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435424    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 14:27:52.441982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435429    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435434    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435439    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435443    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435448    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435453    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435457    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435464    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435470    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435474    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435479    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435484    2566 flags.go:64] FLAG: --enable-server="true"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435489    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435496    2566 flags.go:64] FLAG: --event-burst="100"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435501    2566 flags.go:64] FLAG: --event-qps="50"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435505    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435510    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435514    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435544    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435550    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435556    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435561    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435566    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435586    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435593    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 14:27:52.442659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435597    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435602    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435607    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435612    2566 flags.go:64] FLAG: --feature-gates=""
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435618    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435623    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435627    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435633    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435638    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435643    2566 flags.go:64] FLAG: --help="false"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435647    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435652    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435656    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435662    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435667    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435673    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435677    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435682    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435687    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435691    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435697    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435702    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435706    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435711    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 14:27:52.443254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435715    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435720    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435725    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435729    2566 flags.go:64] FLAG: --lock-file=""
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435734    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435739    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435744    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435753    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435757    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435762    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435767    2566 flags.go:64] FLAG: --logging-format="text"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435771    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435777    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435781    2566 flags.go:64] FLAG: --manifest-url=""
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435785    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435792    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435797    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435803    2566 flags.go:64] FLAG: --max-pods="110"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435808    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435812    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435817    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435822    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435826    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435831    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435836    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 14:27:52.443905 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435847    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435851    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435856    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435861    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435866    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435875    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435879    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435884    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435888    2566 flags.go:64] FLAG: --port="10250"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435893    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435898    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a842c92d84ee6bbb"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435903    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435907    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435912    2566 flags.go:64] FLAG: --register-node="true"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435916    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435922    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435927    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435932    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435936    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435949    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435954    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435959    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435963    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435968    2566 flags.go:64] FLAG: --runonce="false"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435973    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 14:27:52.444511 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435977    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435982    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435987    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435991    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.435996    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436001    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436006    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436010    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436015    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436019    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436023    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436028    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436034    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436038    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436047    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436051    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436056    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436062    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436067    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436072    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436076    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436081    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436086    2566 flags.go:64] FLAG: --v="2"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436092    2566 flags.go:64] FLAG: --version="false"
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436098    2566 flags.go:64] FLAG: --vmodule=""
Sep 10 14:27:52.445131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436103    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.436109    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436291    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436300    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436305    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436310    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436315    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436319    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436326    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436330    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436335    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436340    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436345    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436349    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436353    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436356    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436361    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436365    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436369    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:27:52.445757 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436372    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436377    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436382    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436388    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436392    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436396    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436401    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436405    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436409    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436414    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436418    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436422    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436426    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436431    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436436    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436440    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436444    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436448    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436452    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436456    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:27:52.446242 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436460    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436464    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436468    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436472    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436476    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436480    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436484    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436488    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436492    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436496    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436500    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436504    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436508    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436512    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436517    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436526    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436529    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436534    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436537    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436541    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:27:52.446793 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436545    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436549    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436555    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436559    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436563    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436584    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436589    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436593    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436597    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436601    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436604    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436608    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436613    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436617    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436620    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436625    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436629    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436633    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436636    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436640    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:27:52.447261 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436644    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436648    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436652    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436656    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436660    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436664    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436668    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436675    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436679    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436682    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436686    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436691    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436695    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436699    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436705    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436709    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436713    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436717    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436720    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436724    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:27:52.447743 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436729    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.436732    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.437690    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.445332    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.445347    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.446267    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.446274    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446330    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446336    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446339    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446343    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446346    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446349    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446352    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446355    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:27:52.448211 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446359    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446361    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446364    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446366    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446370    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446372    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446375    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446378    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446380    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446383    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446385    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446388    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446391    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446393    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446396    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446398    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446401    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446404    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446407    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446410    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:27:52.448596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446413    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446416    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446418    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446421    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446424    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446426    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446430    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446433    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446435    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446438    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446440    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446443    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446445    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446448    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446450    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446453    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446455    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446459    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446461    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446464    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:27:52.449399 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446466    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446469    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446471    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446474    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446476    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446479    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446481    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446484    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446486    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446489    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446493    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446497    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446500    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446504    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446507    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446510    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446512    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446515    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446517    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:27:52.450084 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446520    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446522    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446525    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446527    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446530    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446533    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446536    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446540    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446543    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446546    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446548    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446551    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446554    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446558    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446561    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446564    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446581    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446586    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446589    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:27:52.450546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446592    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446595    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446597    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446600    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446603    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446606    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446609    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446612    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446615    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446618    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446620    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446623    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446625    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.446630    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446737    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:27:52.451011 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446743    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446746    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446749    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446754    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446757    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446759    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446763    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446765    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446768    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446771    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446774    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446778    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446780    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446783    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446786    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446789    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446792    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446794    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446797    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446800    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:27:52.451396 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446802    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446805    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446809    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446812    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446814    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446817    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446820    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446822    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446825    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446827    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446830    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446832    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446835    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446837    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446840    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446842    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446845    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446848    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446852    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446854    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:27:52.451889 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446857    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446860    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446862    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446866    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446868    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446870    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446873    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446875    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446878    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446880    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446884    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446887    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446890    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446894    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446897    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446900    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446903    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446905    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446908    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446910    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:27:52.452360 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446913    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446916    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446918    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446920    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446923    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446925    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446928    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446931    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446933    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446936    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446938    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446941    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446943    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446946    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446948    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446951    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446954    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446956    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446959    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446961    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:27:52.452927 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446964    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446966    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446968    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446971    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446973    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446976    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446979    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446981    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446984    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446986    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446989    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446991    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446994    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446996    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.446999    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.447002    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.447004    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:27:52.453408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:52.447006    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:27:52.453840 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.447012    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:27:52.453840 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.447108    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 14:27:52.453840 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.451328    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 14:27:52.453840 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.452451    2566 server.go:1019] "Starting client certificate rotation"
Sep 10 14:27:52.453840 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.452548    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:27:52.453840 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.452597    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:27:52.455454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.455440    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:27:52.456965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.456949    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:27:52.469368 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.469352    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 10 14:27:52.472645 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.472629    2566 log.go:25] "Validated CRI v1 image API"
Sep 10 14:27:52.473513 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.473495    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 14:27:52.478015 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.477994    2566 fs.go:135] Filesystem UUIDs: map[79e2a32a-36bb-43fd-8d55-4770ef674aed:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a10ee548-3943-425b-9fc0-2d8d7e6b64d9:/dev/nvme0n1p4]
Sep 10 14:27:52.478114 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.478013    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 14:27:52.483774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.483756    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:27:52.484230 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.484123    2566 manager.go:216] Machine: {Timestamp:2026-09-10 14:27:52.481692289 +0000 UTC m=+0.346550639 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198128 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d1da146fee43b07535f66e1f0bd39 SystemUUID:ec2d1da1-46fe-e43b-0753-5f66e1f0bd39 BootID:b1e3ee38-1f2a-4037-a892-fab9f414503f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:72:a1:31:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:72:a1:31:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:65:33:f4:8a:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 14:27:52.484230 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.484228    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 14:27:52.484317 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.484310    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 14:27:52.487989 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.487938    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 14:27:52.488377 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.487998    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-162.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 14:27:52.489722 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.489707    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 14:27:52.489788 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.489729    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 14:27:52.489788 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.489740    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:27:52.490434 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.490422    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:27:52.492284 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.492272    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 14:27:52.496883 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.496869    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:27:52.497023 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.497012    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 14:27:52.497083 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.497050    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 14:27:52.497083 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.497065    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 14:27:52.497083 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.497083    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 14:27:52.497200 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.497097    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 10 14:27:52.497200 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.497110    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 14:27:52.497498 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.497484    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:27:52.497564 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.497508    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:27:52.500043 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.500026    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 14:27:52.502920 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.502906    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 14:27:52.502984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.502935    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 14:27:52.503078 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503068    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 14:27:52.503111 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503084    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 14:27:52.503111 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503090    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 14:27:52.503111 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503095    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 14:27:52.503111 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503103    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 14:27:52.503111 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503108    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 14:27:52.503831 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503822    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 14:27:52.503831 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503831    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 14:27:52.503888 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503837    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 14:27:52.503888 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503844    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 14:27:52.503888 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503852    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 14:27:52.503888 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.503859    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 14:27:52.504791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.504781    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 14:27:52.504876 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.504792    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 14:27:52.505785 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.505765    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b2rq5"
Sep 10 14:27:52.507347 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.507326    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-162.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 14:27:52.508099 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.508078    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 14:27:52.508176 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.508154    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 14:27:52.508733 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.508722    2566 server.go:1268] "Started kubelet"
Sep 10 14:27:52.508832 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.508790    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 14:27:52.508864 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.508854    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 14:27:52.508864 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.508847    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 14:27:52.509602 ip-10-0-129-162 systemd[1]: Started Kubernetes Kubelet.
Sep 10 14:27:52.510296 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.510278    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 14:27:52.510762 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.510747    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 14:27:52.512586 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.512552    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b2rq5"
Sep 10 14:27:52.514122 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.513993    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 14:27:52.514381 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.513233    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-162.ec2.internal.18d3fbed0b20160a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-162.ec2.internal,UID:ip-10-0-129-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-162.ec2.internal,},FirstTimestamp:2026-09-10 14:27:52.508691978 +0000 UTC m=+0.373550329,LastTimestamp:2026-09-10 14:27:52.508691978 +0000 UTC m=+0.373550329,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-162.ec2.internal,}"
Sep 10 14:27:52.518311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.518184    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 14:27:52.518902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.518882    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 14:27:52.519000 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.518966    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:52.519641 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519626    2566 factory.go:55] Registering systemd factory
Sep 10 14:27:52.519641 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519642    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 10 14:27:52.519771 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519707    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 14:27:52.519771 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519717    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 14:27:52.519771 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519730    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 14:27:52.519871 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519800    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 14:27:52.519871 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519809    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 14:27:52.519871 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519835    2566 factory.go:153] Registering CRI-O factory
Sep 10 14:27:52.519871 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519843    2566 factory.go:223] Registration of the crio container factory successfully
Sep 10 14:27:52.520031 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519895    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 14:27:52.520031 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519911    2566 factory.go:103] Registering Raw factory
Sep 10 14:27:52.520031 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.519922    2566 manager.go:1196] Started watching for new ooms in manager
Sep 10 14:27:52.520251 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.520239    2566 manager.go:349] Starting recovery of all containers
Sep 10 14:27:52.520802 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.520774    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 14:27:52.520897 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.520875    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 14:27:52.531844 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.531826    2566 manager.go:354] Recovery completed
Sep 10 14:27:52.537304 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.537290    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:27:52.539175 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.539160    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:27:52.539254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.539188    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:27:52.539254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.539200    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:27:52.539774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.539761    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 14:27:52.539774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.539772    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 14:27:52.539881 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.539785    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:27:52.539881 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.539823    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:27:52.540765 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.540751    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:27:52.540828 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.540777    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:27:52.540828 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.540786    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.619824    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.662761    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.664766    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.664804    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.664824    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.664862    2566 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.666979    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.683557    2566 policy_none.go:49] "None policy: Start"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.683594    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 14:27:52.684774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.683609    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 14:27:52.685632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.685620    2566 policy_none.go:47] "Start"
Sep 10 14:27:52.720019 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.719995    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:52.720097 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.720090    2566 manager.go:337] "Starting Device Plugin manager"
Sep 10 14:27:52.720140 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.720127    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 14:27:52.720172 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.720144    2566 server.go:87] "Starting device plugin registration server"
Sep 10 14:27:52.720473 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.720459    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 14:27:52.720539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.720473    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 14:27:52.720613 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.720594    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 14:27:52.720694 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.720677    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 14:27:52.720694 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.720690    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 14:27:52.721249 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.721226    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 14:27:52.721343 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.721283    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:52.726854 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.726838    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-162.ec2.internal\" not found" node="ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.765992 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.765966    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal"]
Sep 10 14:27:52.766089 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.766045    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:27:52.766948 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.766933    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:27:52.767032 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.766965    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:27:52.767032 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.766980    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:27:52.769295 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.769280    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:27:52.769401 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.769385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.769446 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.769418    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:27:52.770041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.770026    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:27:52.770106 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.770046    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:27:52.770106 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.770063    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:27:52.770106 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.770081    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:27:52.770217 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.770050    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:27:52.770217 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.770141    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:27:52.772723 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.772708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.772788 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.772736    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:27:52.773444 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.773429    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:27:52.773528 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.773455    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:27:52.773528 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.773473    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:27:52.797061 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.797042    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-162.ec2.internal\" not found" node="ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.801325 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.801309    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-162.ec2.internal\" not found" node="ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.820540 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.820519    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:27:52.821221 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.821195    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:27:52.821221 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.821223    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:27:52.821361 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.821236    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:27:52.821361 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.821261    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.827741 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.827727    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.827807 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.827745    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-162.ec2.internal\": node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:52.844687 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.844673    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:52.920814 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.920781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ca4afa319417de5b8cf82bebe639656-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal\" (UID: \"0ca4afa319417de5b8cf82bebe639656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.920814 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.920816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ca4afa319417de5b8cf82bebe639656-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal\" (UID: \"0ca4afa319417de5b8cf82bebe639656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.920928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:52.920833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0c71d27d7840b6e3fe5cea3210406d3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-162.ec2.internal\" (UID: \"d0c71d27d7840b6e3fe5cea3210406d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:52.944910 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:52.944864    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:53.021543 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:53.021509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ca4afa319417de5b8cf82bebe639656-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal\" (UID: \"0ca4afa319417de5b8cf82bebe639656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:53.021543 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:53.021547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ca4afa319417de5b8cf82bebe639656-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal\" (UID: \"0ca4afa319417de5b8cf82bebe639656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:53.021696 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:53.021564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0c71d27d7840b6e3fe5cea3210406d3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-162.ec2.internal\" (UID: \"d0c71d27d7840b6e3fe5cea3210406d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:53.021696 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:53.021623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0c71d27d7840b6e3fe5cea3210406d3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-162.ec2.internal\" (UID: \"d0c71d27d7840b6e3fe5cea3210406d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:53.021696 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:53.021622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0ca4afa319417de5b8cf82bebe639656-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal\" (UID: \"0ca4afa319417de5b8cf82bebe639656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:53.021696 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:53.021670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ca4afa319417de5b8cf82bebe639656-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal\" (UID: \"0ca4afa319417de5b8cf82bebe639656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:53.045626 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:53.045606    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:53.099776 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:53.099757    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:53.104621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:53.104511    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:53.146355 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:53.146328    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:53.246944 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:53.246861    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:54.786433 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:54.786396    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:54.809871 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:54.809835    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:27:54.887348 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:54.887315    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-162.ec2.internal\" not found"
Sep 10 14:27:54.891471 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:54.891455    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 14:27:54.891614 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:54.891592    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:27:54.891666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:54.891592    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:27:54.958614 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:54.958544    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 14:22:53 +0000 UTC" deadline="2028-05-18 22:51:43.707226806 +0000 UTC"
Sep 10 14:27:54.958614 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:54.958569    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14792h23m48.748660545s"
Sep 10 14:27:54.959742 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:54.959726    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:27:55.058488 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.058458    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:55.070522 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.070505    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:27:55.072048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.072036    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal"
Sep 10 14:27:55.080217 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.080203    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:27:55.301200 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:55.301150    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca4afa319417de5b8cf82bebe639656.slice/crio-4b287fc5e4bc3789c9a38315340cf13de404169b6835268dde4d1ec01ae0b0f1 WatchSource:0}: Error finding container 4b287fc5e4bc3789c9a38315340cf13de404169b6835268dde4d1ec01ae0b0f1: Status 404 returned error can't find the container with id 4b287fc5e4bc3789c9a38315340cf13de404169b6835268dde4d1ec01ae0b0f1
Sep 10 14:27:55.301505 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:55.301481    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c71d27d7840b6e3fe5cea3210406d3.slice/crio-4d7a723c5e06329a51c63b85eb0797d401e0c5946c7a2a6e7cf40bbd59d9cdfe WatchSource:0}: Error finding container 4d7a723c5e06329a51c63b85eb0797d401e0c5946c7a2a6e7cf40bbd59d9cdfe: Status 404 returned error can't find the container with id 4d7a723c5e06329a51c63b85eb0797d401e0c5946c7a2a6e7cf40bbd59d9cdfe
Sep 10 14:27:55.306731 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.306717    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:27:55.537398 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.537373    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:27:55.937364 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.937332    2566 apiserver.go:52] "Watching apiserver"
Sep 10 14:27:55.945957 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.945932    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 14:27:55.946325 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.946302    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8tb9v","kube-system/konnectivity-agent-sxhmb","kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal","openshift-dns/node-resolver-v5n69","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal","openshift-multus/multus-additional-cni-plugins-bpg6r","openshift-multus/network-metrics-daemon-qfbg6","openshift-network-operator/iptables-alerter-k6254","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp","openshift-cluster-node-tuning-operator/tuned-tc6th","openshift-image-registry/node-ca-fvv6t","openshift-multus/multus-jr8gf","openshift-network-diagnostics/network-check-target-c7jb9","openshift-ovn-kubernetes/ovnkube-node-hbrcr"]
Sep 10 14:27:55.948789 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.948768    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:55.948881 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:55.948834    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:27:55.950968 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.950949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:55.953350 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.953018    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-75k5q\""
Sep 10 14:27:55.953350 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.953050    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 14:27:55.953350 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.953105    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 14:27:55.953350 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.953223    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 14:27:55.953350 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.953292    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.955223 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.955205    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 14:27:55.955635 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.955375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:55.955635 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.955421    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 14:27:55.955779 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.955744    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 14:27:55.955834 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.955778    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6lsg4\""
Sep 10 14:27:55.955834 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.955804    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 14:27:55.956661 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.956597    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 14:27:55.956661 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.956620    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 14:27:55.957577 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.957558    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 14:27:55.957663 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.957578    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 14:27:55.957663 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.957635    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 14:27:55.957784 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.957636    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-njw85\""
Sep 10 14:27:55.958053 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.958035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:55.959847 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.959795    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:27:55.960048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.960031    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dqdvj\""
Sep 10 14:27:55.960566 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.960540    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 14:27:55.960566 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.960556    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 14:27:55.960995 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.960975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:55.962872 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.962853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 14:27:55.964095 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.964075    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zp2m4\""
Sep 10 14:27:55.964395 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.964376    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 14:27:55.964762 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.964742    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 14:27:55.965033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.965015    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 14:27:55.965526 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.965509    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 14:27:55.965842 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.965826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:55.967318 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.966965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:27:55.967318 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:55.967082    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:27:55.967318 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07145279-d28b-4f9a-9a9b-d7b763413fd1-iptables-alerter-script\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:55.967491 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967348    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzsd\" (UniqueName: \"kubernetes.io/projected/fe629083-72c1-416f-a8ab-38fe211e7598-kube-api-access-kmzsd\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:55.967491 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/532db11a-6291-4687-8d65-e71384f16400-serviceca\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:55.967569 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-slash\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.967616 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-run-netns\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.967668 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.967739 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-cni-bin\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.967790 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-ovn\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.967790 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-node-log\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.967851 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:55.967902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:55.967940 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:55.967988 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.967949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovnkube-script-lib\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968038 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9769n\" (UniqueName: \"kubernetes.io/projected/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-kube-api-access-9769n\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:55.968070 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/532db11a-6291-4687-8d65-e71384f16400-host\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:55.968110 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovnkube-config\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968164 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:55.968198 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07145279-d28b-4f9a-9a9b-d7b763413fd1-host-slash\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:55.968236 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7b4\" (UniqueName: \"kubernetes.io/projected/352316f0-970c-43ad-befe-ffeaaad7e582-kube-api-access-db7b4\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:55.968282 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:55.968320 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-systemd-units\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-systemd\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968397 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968383    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 14:27:55.968433 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352316f0-970c-43ad-befe-ffeaaad7e582-tmp-dir\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:55.968484 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-system-cni-dir\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:55.968517 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968504    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cnibin\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:55.968595 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwsc\" (UniqueName: \"kubernetes.io/projected/532db11a-6291-4687-8d65-e71384f16400-kube-api-access-gcwsc\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:55.968630 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-cni-netd\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968681 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovn-node-metrics-cert\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968760 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkwx\" (UniqueName: \"kubernetes.io/projected/055abf0a-a771-42fa-aa0a-be5cd2b73692-kube-api-access-hrkwx\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968816 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-var-lib-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968868 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-etc-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.968976 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467vb\" (UniqueName: \"kubernetes.io/projected/07145279-d28b-4f9a-9a9b-d7b763413fd1-kube-api-access-467vb\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:55.968976 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-kubelet\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.969069 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-log-socket\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.969069 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.969043    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j5sdw\""
Sep 10 14:27:55.969457 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.969434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.969585 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.969566    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 14:27:55.969650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.969590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-env-overrides\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:55.969650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.969622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/352316f0-970c-43ad-befe-ffeaaad7e582-hosts-file\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:55.969774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.968782    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 14:27:55.969899 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.969881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:27:55.971924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.971905    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:27:55.972169 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.972150    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rph4b\""
Sep 10 14:27:55.972251 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.972234    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 14:27:55.972330 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.972298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:55.972402 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.972340    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 14:27:55.974253 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.974237    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bzc9v\""
Sep 10 14:27:55.974420 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.974404    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:27:55.974531 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.974515    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 14:27:55.974782 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.974763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:55.974862 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:55.974839    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:27:55.977189 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.977159    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:55.979135 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.979114    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 14:27:55.979300 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.979282    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4qd9v\""
Sep 10 14:27:55.993250 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:55.993228    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7vjqr"
Sep 10 14:27:56.001172 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.001151    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7vjqr"
Sep 10 14:27:56.060068 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.060029    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 14:27:56.070148 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-etc-kubernetes\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.070268 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-sys\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.070268 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-system-cni-dir\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.070268 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cnibin\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.070268 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-cni-netd\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9x8\" (UniqueName: \"kubernetes.io/projected/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-kube-api-access-kg9x8\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5gc\" (UniqueName: \"kubernetes.io/projected/65fab78a-e493-4fc4-b491-bc6bd32e73da-kube-api-access-5w5gc\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-var-lib-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-etc-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-socket-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-registration-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-dbus\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysconfig\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.070510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-log-socket\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-etc-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-kubelet\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzsd\" (UniqueName: \"kubernetes.io/projected/fe629083-72c1-416f-a8ab-38fe211e7598-kube-api-access-kmzsd\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/532db11a-6291-4687-8d65-e71384f16400-serviceca\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-slash\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-run-netns\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-log-socket\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-cni-bin\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-hostroot\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-cni-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.071041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cnibin\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47492653-52e9-41d3-94e3-616fef6d8780-agent-certs\") pod \"konnectivity-agent-sxhmb\" (UID: \"47492653-52e9-41d3-94e3-616fef6d8780\") " pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-system-cni-dir\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.070977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-cni-netd\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-cni-bin\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-slash\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-device-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-var-lib-openvswitch\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-host\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-run-netns\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-tuned\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-systemd\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.071965 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-systemd\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.072755 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352316f0-970c-43ad-befe-ffeaaad7e582-tmp-dir\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:56.072755 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.071994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.073102 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.073076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.073349 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.072030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.073436 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.073377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnlp\" (UniqueName: \"kubernetes.io/projected/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-kube-api-access-ptnlp\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.073607 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.073525    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352316f0-970c-43ad-befe-ffeaaad7e582-tmp-dir\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:56.073607 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.073580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/532db11a-6291-4687-8d65-e71384f16400-serviceca\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:56.073782 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.073672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysctl-conf\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.073834 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.073778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwsc\" (UniqueName: \"kubernetes.io/projected/532db11a-6291-4687-8d65-e71384f16400-kube-api-access-gcwsc\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:56.073882 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.073841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovn-node-metrics-cert\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.073882 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.073872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkwx\" (UniqueName: \"kubernetes.io/projected/055abf0a-a771-42fa-aa0a-be5cd2b73692-kube-api-access-hrkwx\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.074252 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47492653-52e9-41d3-94e3-616fef6d8780-konnectivity-ca\") pod \"konnectivity-agent-sxhmb\" (UID: \"47492653-52e9-41d3-94e3-616fef6d8780\") " pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:27:56.074347 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-kubernetes\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.074347 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysctl-d\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.074347 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65fab78a-e493-4fc4-b491-bc6bd32e73da-tmp\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.074485 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-kubelet-config\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.074485 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-run\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.074569 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467vb\" (UniqueName: \"kubernetes.io/projected/07145279-d28b-4f9a-9a9b-d7b763413fd1-kube-api-access-467vb\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:56.074614 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-kubelet\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.074663 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-env-overrides\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.074720 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/352316f0-970c-43ad-befe-ffeaaad7e582-hosts-file\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:56.074772 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:27:56.074819 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07145279-d28b-4f9a-9a9b-d7b763413fd1-iptables-alerter-script\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:56.074860 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-socket-dir-parent\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.074970 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-conf-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.075034 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-host-kubelet\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.075034 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074988    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 14:27:56.075034 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.074994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-ovn\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.075206 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-node-log\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.075206 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:56.075206 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovnkube-script-lib\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.075330 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.075378 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-multus-certs\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.075378 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075358    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-modprobe-d\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.075467 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9769n\" (UniqueName: \"kubernetes.io/projected/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-kube-api-access-9769n\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.075467 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/532db11a-6291-4687-8d65-e71384f16400-host\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:56.075467 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovnkube-config\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.075467 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-env-overrides\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.075638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-sys-fs\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.075638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-daemon-config\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.075638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.075538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-systemd\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.076265 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovnkube-config\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.076363 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-lib-modules\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.076363 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07145279-d28b-4f9a-9a9b-d7b763413fd1-host-slash\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:56.076477 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7b4\" (UniqueName: \"kubernetes.io/projected/352316f0-970c-43ad-befe-ffeaaad7e582-kube-api-access-db7b4\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:56.076477 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovnkube-script-lib\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.076477 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07145279-d28b-4f9a-9a9b-d7b763413fd1-host-slash\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:56.076477 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/352316f0-970c-43ad-befe-ffeaaad7e582-hosts-file\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:56.076749 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/532db11a-6291-4687-8d65-e71384f16400-host\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:56.077409 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.076912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-run-ovn\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.077409 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.077048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-system-cni-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.077613 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.077490    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07145279-d28b-4f9a-9a9b-d7b763413fd1-iptables-alerter-script\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:56.077738 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.077551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-node-log\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.077738 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.077653    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:27:56.077738 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.077715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-cnibin\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.077896 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.077772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-cni-binary-copy\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.077896 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.077851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-cni-bin\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.077992 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.077896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-var-lib-kubelet\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.077992 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.077951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-systemd-units\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.078089 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.078033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.078139 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.078105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/055abf0a-a771-42fa-aa0a-be5cd2b73692-systemd-units\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.078139 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.078117    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-k8s-cni-cncf-io\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.078233 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.078175    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs podName:fe629083-72c1-416f-a8ab-38fe211e7598 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:56.578141282 +0000 UTC m=+3.003878698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs") pod "network-metrics-daemon-qfbg6" (UID: "fe629083-72c1-416f-a8ab-38fe211e7598") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:27:56.078233 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.078210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-netns\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.078339 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.078240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-cni-multus\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.081574 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.081515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/055abf0a-a771-42fa-aa0a-be5cd2b73692-ovn-node-metrics-cert\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.082217 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.082151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzsd\" (UniqueName: \"kubernetes.io/projected/fe629083-72c1-416f-a8ab-38fe211e7598-kube-api-access-kmzsd\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:56.083386 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.083341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-467vb\" (UniqueName: \"kubernetes.io/projected/07145279-d28b-4f9a-9a9b-d7b763413fd1-kube-api-access-467vb\") pod \"iptables-alerter-k6254\" (UID: \"07145279-d28b-4f9a-9a9b-d7b763413fd1\") " pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:56.084675 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.084650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwsc\" (UniqueName: \"kubernetes.io/projected/532db11a-6291-4687-8d65-e71384f16400-kube-api-access-gcwsc\") pod \"node-ca-fvv6t\" (UID: \"532db11a-6291-4687-8d65-e71384f16400\") " pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:56.085107 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.085085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9769n\" (UniqueName: \"kubernetes.io/projected/538b5488-3d52-4ae3-8ffc-5e3be2ac32de-kube-api-access-9769n\") pod \"multus-additional-cni-plugins-bpg6r\" (UID: \"538b5488-3d52-4ae3-8ffc-5e3be2ac32de\") " pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.085243 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.085187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkwx\" (UniqueName: \"kubernetes.io/projected/055abf0a-a771-42fa-aa0a-be5cd2b73692-kube-api-access-hrkwx\") pod \"ovnkube-node-hbrcr\" (UID: \"055abf0a-a771-42fa-aa0a-be5cd2b73692\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.086172 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.086134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7b4\" (UniqueName: \"kubernetes.io/projected/352316f0-970c-43ad-befe-ffeaaad7e582-kube-api-access-db7b4\") pod \"node-resolver-v5n69\" (UID: \"352316f0-970c-43ad-befe-ffeaaad7e582\") " pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:56.110522 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.110475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal" event={"ID":"0ca4afa319417de5b8cf82bebe639656","Type":"ContainerStarted","Data":"4b287fc5e4bc3789c9a38315340cf13de404169b6835268dde4d1ec01ae0b0f1"}
Sep 10 14:27:56.112340 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.112316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal" event={"ID":"d0c71d27d7840b6e3fe5cea3210406d3","Type":"ContainerStarted","Data":"4d7a723c5e06329a51c63b85eb0797d401e0c5946c7a2a6e7cf40bbd59d9cdfe"}
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-multus-certs\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-modprobe-d\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-sys-fs\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-daemon-config\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.179675    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-multus-certs\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.179768    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret podName:6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:56.679736872 +0000 UTC m=+3.105474284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret") pod "global-pull-secret-syncer-8tb9v" (UID: "6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-systemd\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179680    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-systemd\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-lib-modules\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-system-cni-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-cnibin\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-modprobe-d\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-cni-binary-copy\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-cni-bin\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.180901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-sys-fs\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-var-lib-kubelet\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.179981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-k8s-cni-cncf-io\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-system-cni-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-netns\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-cni-multus\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-etc-kubernetes\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180106    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-lib-modules\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-sys\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9x8\" (UniqueName: \"kubernetes.io/projected/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-kube-api-access-kg9x8\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-sys\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5gc\" (UniqueName: \"kubernetes.io/projected/65fab78a-e493-4fc4-b491-bc6bd32e73da-kube-api-access-5w5gc\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-cnibin\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-socket-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-registration-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-dbus\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysconfig\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.181827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-kubelet\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-hostroot\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-cni-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47492653-52e9-41d3-94e3-616fef6d8780-agent-certs\") pod \"konnectivity-agent-sxhmb\" (UID: \"47492653-52e9-41d3-94e3-616fef6d8780\") " pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-device-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-host\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-tuned\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-daemon-config\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnlp\" (UniqueName: \"kubernetes.io/projected/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-kube-api-access-ptnlp\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysctl-conf\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-cni-binary-copy\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180686    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-cni-bin\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysctl-conf\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-var-lib-kubelet\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-kubelet\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.182621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-k8s-cni-cncf-io\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-run-netns\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-host-var-lib-cni-multus\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-hostroot\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.180945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-etc-kubernetes\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-host\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-cni-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-socket-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-registration-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-dbus\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysconfig\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47492653-52e9-41d3-94e3-616fef6d8780-konnectivity-ca\") pod \"konnectivity-agent-sxhmb\" (UID: \"47492653-52e9-41d3-94e3-616fef6d8780\") " pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-kubernetes\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysctl-d\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65fab78a-e493-4fc4-b491-bc6bd32e73da-tmp\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181369    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-device-dir\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-kubelet-config\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-kubelet-config\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.183454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-run\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-socket-dir-parent\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-conf-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-conf-dir\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-run\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-sysctl-d\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-multus-socket-dir-parent\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-kubernetes\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.181905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47492653-52e9-41d3-94e3-616fef6d8780-konnectivity-ca\") pod \"konnectivity-agent-sxhmb\" (UID: \"47492653-52e9-41d3-94e3-616fef6d8780\") " pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.183340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65fab78a-e493-4fc4-b491-bc6bd32e73da-etc-tuned\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.184311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.183676    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47492653-52e9-41d3-94e3-616fef6d8780-agent-certs\") pod \"konnectivity-agent-sxhmb\" (UID: \"47492653-52e9-41d3-94e3-616fef6d8780\") " pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:27:56.187018 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.186971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65fab78a-e493-4fc4-b491-bc6bd32e73da-tmp\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.189463 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.189409    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:27:56.189463 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.189431    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:27:56.189463 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.189444    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6ds6k for pod openshift-network-diagnostics/network-check-target-c7jb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:27:56.189648 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.189522    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k podName:8abd9fae-29f2-42aa-a064-3c1928c21286 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:56.689489486 +0000 UTC m=+3.115226892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6ds6k" (UniqueName: "kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k") pod "network-check-target-c7jb9" (UID: "8abd9fae-29f2-42aa-a064-3c1928c21286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:27:56.192351 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.192312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnlp\" (UniqueName: \"kubernetes.io/projected/7dc378d9-85fc-47e5-93bd-5d7a31b5257d-kube-api-access-ptnlp\") pod \"aws-ebs-csi-driver-node-5fqfp\" (UID: \"7dc378d9-85fc-47e5-93bd-5d7a31b5257d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.193288 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.192886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9x8\" (UniqueName: \"kubernetes.io/projected/3d97cd1e-f9fb-4f92-ab4c-2068152f5d01-kube-api-access-kg9x8\") pod \"multus-jr8gf\" (UID: \"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01\") " pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.193288 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.193253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5gc\" (UniqueName: \"kubernetes.io/projected/65fab78a-e493-4fc4-b491-bc6bd32e73da-kube-api-access-5w5gc\") pod \"tuned-tc6th\" (UID: \"65fab78a-e493-4fc4-b491-bc6bd32e73da\") " pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.263220 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.263183    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fvv6t"
Sep 10 14:27:56.274708 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.274670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:27:56.282492 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.282468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5n69"
Sep 10 14:27:56.288482 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.288452    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6254"
Sep 10 14:27:56.295862 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.295712    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bpg6r"
Sep 10 14:27:56.302477 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.302451    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp"
Sep 10 14:27:56.309640 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.309620    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:27:56.316858 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.316713    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tc6th"
Sep 10 14:27:56.321670 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.321648    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jr8gf"
Sep 10 14:27:56.401795 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.401764    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:27:56.584822 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.584731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:56.584973 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.584913    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:27:56.585026 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.584994    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs podName:fe629083-72c1-416f-a8ab-38fe211e7598 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:57.584973752 +0000 UTC m=+4.010711153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs") pod "network-metrics-daemon-qfbg6" (UID: "fe629083-72c1-416f-a8ab-38fe211e7598") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:27:56.686106 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.686072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:56.686288 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.686195    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:27:56.686288 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.686268    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret podName:6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:57.686254169 +0000 UTC m=+4.111991566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret") pod "global-pull-secret-syncer-8tb9v" (UID: "6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:27:56.787285 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:56.787254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:27:56.787482 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.787461    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:27:56.787531 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.787489    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:27:56.787531 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.787503    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6ds6k for pod openshift-network-diagnostics/network-check-target-c7jb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:27:56.787605 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:56.787565    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k podName:8abd9fae-29f2-42aa-a064-3c1928c21286 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:57.787547414 +0000 UTC m=+4.213284826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ds6k" (UniqueName: "kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k") pod "network-check-target-c7jb9" (UID: "8abd9fae-29f2-42aa-a064-3c1928c21286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:27:57.004457 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:57.004372    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:22:55 +0000 UTC" deadline="2028-03-02 14:35:56.07226114 +0000 UTC"
Sep 10 14:27:57.004457 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:57.004404    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12936h7m59.067859836s"
Sep 10 14:27:57.105055 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:57.105020    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:57.105215 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:57.105053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:27:57.105215 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.105131    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:27:57.105287 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.105232    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:27:57.213501 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:57.213462    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07145279_d28b_4f9a_9a9b_d7b763413fd1.slice/crio-e03d080bc42a001d4b275811c4244c97ceccb13697e42c5149214b9b7fe069f4 WatchSource:0}: Error finding container e03d080bc42a001d4b275811c4244c97ceccb13697e42c5149214b9b7fe069f4: Status 404 returned error can't find the container with id e03d080bc42a001d4b275811c4244c97ceccb13697e42c5149214b9b7fe069f4
Sep 10 14:27:57.215066 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:57.214996    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc378d9_85fc_47e5_93bd_5d7a31b5257d.slice/crio-d001c93157be88c3a59c8ed30486dccd4a970f0a1e399c02f7498cbe35a6ef3e WatchSource:0}: Error finding container d001c93157be88c3a59c8ed30486dccd4a970f0a1e399c02f7498cbe35a6ef3e: Status 404 returned error can't find the container with id d001c93157be88c3a59c8ed30486dccd4a970f0a1e399c02f7498cbe35a6ef3e
Sep 10 14:27:57.216275 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:57.216059    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538b5488_3d52_4ae3_8ffc_5e3be2ac32de.slice/crio-a0bc37f3ae04786e6d0e9c044b3386269c824618e00fef49469075254204d6e9 WatchSource:0}: Error finding container a0bc37f3ae04786e6d0e9c044b3386269c824618e00fef49469075254204d6e9: Status 404 returned error can't find the container with id a0bc37f3ae04786e6d0e9c044b3386269c824618e00fef49469075254204d6e9
Sep 10 14:27:57.216354 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:57.216289    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352316f0_970c_43ad_befe_ffeaaad7e582.slice/crio-5b7e7a88354cfaceb7f940323a975f574d38dad1e7b9a8cf7aa04ea9ba0ed21d WatchSource:0}: Error finding container 5b7e7a88354cfaceb7f940323a975f574d38dad1e7b9a8cf7aa04ea9ba0ed21d: Status 404 returned error can't find the container with id 5b7e7a88354cfaceb7f940323a975f574d38dad1e7b9a8cf7aa04ea9ba0ed21d
Sep 10 14:27:57.219638 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:57.219615    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532db11a_6291_4687_8d65_e71384f16400.slice/crio-9d9267996622cb8d024504f15957645e1b6c7a3ca7460a74eb64938b9c061e5d WatchSource:0}: Error finding container 9d9267996622cb8d024504f15957645e1b6c7a3ca7460a74eb64938b9c061e5d: Status 404 returned error can't find the container with id 9d9267996622cb8d024504f15957645e1b6c7a3ca7460a74eb64938b9c061e5d
Sep 10 14:27:57.220658 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:27:57.220625    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47492653_52e9_41d3_94e3_616fef6d8780.slice/crio-e45e1b8ab414d77af715eb10381034d6dc9d14e065be7d2d7515464f504e58dd WatchSource:0}: Error finding container e45e1b8ab414d77af715eb10381034d6dc9d14e065be7d2d7515464f504e58dd: Status 404 returned error can't find the container with id e45e1b8ab414d77af715eb10381034d6dc9d14e065be7d2d7515464f504e58dd
Sep 10 14:27:57.592008 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:57.591918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:57.592177 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.592074    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:27:57.592177 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.592141    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs podName:fe629083-72c1-416f-a8ab-38fe211e7598 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:59.592124673 +0000 UTC m=+6.017862083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs") pod "network-metrics-daemon-qfbg6" (UID: "fe629083-72c1-416f-a8ab-38fe211e7598") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:27:57.693784 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:57.693120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:57.693784 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.693301    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:27:57.693784 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.693381    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret podName:6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:59.693362004 +0000 UTC m=+6.119099414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret") pod "global-pull-secret-syncer-8tb9v" (UID: "6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:27:57.764956 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:57.764745    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:27:57.794645 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:57.794038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:27:57.794645 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.794214    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:27:57.794645 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.794234    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:27:57.794645 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.794246    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6ds6k for pod openshift-network-diagnostics/network-check-target-c7jb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:27:57.794645 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:57.794302    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k podName:8abd9fae-29f2-42aa-a064-3c1928c21286 nodeName:}" failed. No retries permitted until 2026-09-10 14:27:59.794283254 +0000 UTC m=+6.220020665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ds6k" (UniqueName: "kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k") pod "network-check-target-c7jb9" (UID: "8abd9fae-29f2-42aa-a064-3c1928c21286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:27:58.005011 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.004855    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:22:55 +0000 UTC" deadline="2028-06-02 20:58:24.113185234 +0000 UTC"
Sep 10 14:27:58.005011 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.004918    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15150h30m26.108270034s"
Sep 10 14:27:58.104975 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.104946    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:58.105149 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:58.105081    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:27:58.122229 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.122106    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"82af016aad281d861f08145068faceeb0f17602a97f6815ee31ed10e71fd66ef"}
Sep 10 14:27:58.139331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.139301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jr8gf" event={"ID":"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01","Type":"ContainerStarted","Data":"9f78c64c66c23b728e0ba2723cdf3c549c61252ee6f905571ce1aaf10000f696"}
Sep 10 14:27:58.149299 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.149239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tc6th" event={"ID":"65fab78a-e493-4fc4-b491-bc6bd32e73da","Type":"ContainerStarted","Data":"f3017cc870a60b53733d1582b67a1246f078b3fa4439941644af7bfd21004367"}
Sep 10 14:27:58.160055 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.159790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxhmb" event={"ID":"47492653-52e9-41d3-94e3-616fef6d8780","Type":"ContainerStarted","Data":"e45e1b8ab414d77af715eb10381034d6dc9d14e065be7d2d7515464f504e58dd"}
Sep 10 14:27:58.179711 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.179662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fvv6t" event={"ID":"532db11a-6291-4687-8d65-e71384f16400","Type":"ContainerStarted","Data":"9d9267996622cb8d024504f15957645e1b6c7a3ca7460a74eb64938b9c061e5d"}
Sep 10 14:27:58.188254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.188228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5n69" event={"ID":"352316f0-970c-43ad-befe-ffeaaad7e582","Type":"ContainerStarted","Data":"5b7e7a88354cfaceb7f940323a975f574d38dad1e7b9a8cf7aa04ea9ba0ed21d"}
Sep 10 14:27:58.191418 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.191394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" event={"ID":"538b5488-3d52-4ae3-8ffc-5e3be2ac32de","Type":"ContainerStarted","Data":"a0bc37f3ae04786e6d0e9c044b3386269c824618e00fef49469075254204d6e9"}
Sep 10 14:27:58.197417 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.196869    2566 generic.go:356] "Generic (PLEG): container finished" podID="0ca4afa319417de5b8cf82bebe639656" containerID="94fbffb502e1f138198ced3cde70d1b6c266ec32c3de96033ce56eec314e74cf" exitCode=0
Sep 10 14:27:58.197417 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.197115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal" event={"ID":"0ca4afa319417de5b8cf82bebe639656","Type":"ContainerDied","Data":"94fbffb502e1f138198ced3cde70d1b6c266ec32c3de96033ce56eec314e74cf"}
Sep 10 14:27:58.210510 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.210468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal" event={"ID":"d0c71d27d7840b6e3fe5cea3210406d3","Type":"ContainerStarted","Data":"2aec9a1bdefa93e5f68b7ae829d025043e69a06643def66e2029c78fb115a37e"}
Sep 10 14:27:58.217593 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.217567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp" event={"ID":"7dc378d9-85fc-47e5-93bd-5d7a31b5257d","Type":"ContainerStarted","Data":"d001c93157be88c3a59c8ed30486dccd4a970f0a1e399c02f7498cbe35a6ef3e"}
Sep 10 14:27:58.222243 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:58.222218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6254" event={"ID":"07145279-d28b-4f9a-9a9b-d7b763413fd1","Type":"ContainerStarted","Data":"e03d080bc42a001d4b275811c4244c97ceccb13697e42c5149214b9b7fe069f4"}
Sep 10 14:27:59.105052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:59.105019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:27:59.105687 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.105139    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:27:59.105687 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:59.105220    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:59.105687 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.105303    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:27:59.241183 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:59.241103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal" event={"ID":"0ca4afa319417de5b8cf82bebe639656","Type":"ContainerStarted","Data":"f14ed55f6797eb718f31539e96e2c2dd5db76c32f2b7b09573d0f4536d9a3590"}
Sep 10 14:27:59.256296 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:59.256228    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-162.ec2.internal" podStartSLOduration=4.256205976 podStartE2EDuration="4.256205976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:27:58.229413819 +0000 UTC m=+4.655151238" watchObservedRunningTime="2026-09-10 14:27:59.256205976 +0000 UTC m=+5.681943418"
Sep 10 14:27:59.606771 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:59.606726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:27:59.606962 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.606884    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:27:59.606962 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.606962    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs podName:fe629083-72c1-416f-a8ab-38fe211e7598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:03.606942625 +0000 UTC m=+10.032680044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs") pod "network-metrics-daemon-qfbg6" (UID: "fe629083-72c1-416f-a8ab-38fe211e7598") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:27:59.707370 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:59.707334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:27:59.707531 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.707485    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:27:59.707611 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.707543    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret podName:6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:03.70752622 +0000 UTC m=+10.133263619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret") pod "global-pull-secret-syncer-8tb9v" (UID: "6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:27:59.808012 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:27:59.807974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:27:59.808171 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.808127    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:27:59.808171 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.808159    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:27:59.808171 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.808172    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6ds6k for pod openshift-network-diagnostics/network-check-target-c7jb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:27:59.808339 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:27:59.808227    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k podName:8abd9fae-29f2-42aa-a064-3c1928c21286 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:03.808210091 +0000 UTC m=+10.233947492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ds6k" (UniqueName: "kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k") pod "network-check-target-c7jb9" (UID: "8abd9fae-29f2-42aa-a064-3c1928c21286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:28:00.104684 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:00.104599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:00.104849 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:00.104749    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:01.104433 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:01.104402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:01.104908 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:01.104516    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:01.104908 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:01.104719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:01.104908 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:01.104834    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:02.106392 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:02.105849    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:02.106392 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:02.105990    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:03.104990 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:03.104910    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:03.104990 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:03.104959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:03.105196 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.105081    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:03.105254 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.105216    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:03.641642 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:03.641604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:03.642131 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.641800    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:28:03.642131 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.641881    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs podName:fe629083-72c1-416f-a8ab-38fe211e7598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:11.641858472 +0000 UTC m=+18.067595885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs") pod "network-metrics-daemon-qfbg6" (UID: "fe629083-72c1-416f-a8ab-38fe211e7598") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:28:03.742741 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:03.742705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:03.742930 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.742819    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:28:03.742930 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.742891    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret podName:6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:11.742871839 +0000 UTC m=+18.168609241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret") pod "global-pull-secret-syncer-8tb9v" (UID: "6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:28:03.843818 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:03.843780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:03.844013 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.843996    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:28:03.844075 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.844022    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:28:03.844075 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.844034    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6ds6k for pod openshift-network-diagnostics/network-check-target-c7jb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:28:03.844186 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:03.844091    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k podName:8abd9fae-29f2-42aa-a064-3c1928c21286 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:11.844074802 +0000 UTC m=+18.269812213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ds6k" (UniqueName: "kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k") pod "network-check-target-c7jb9" (UID: "8abd9fae-29f2-42aa-a064-3c1928c21286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:28:04.105580 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:04.105497    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:04.105757 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:04.105611    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:05.105274 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:05.104613    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:05.105274 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:05.104755    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:05.105274 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:05.104825    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:05.105274 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:05.104907    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:06.104941 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:06.104909    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:06.105107 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:06.105040    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:07.105041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:07.105012    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:07.105382 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:07.105026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:07.105382 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:07.105121    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:07.105382 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:07.105236    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:08.104568 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:08.104534    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:08.104744 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:08.104672    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:09.104172 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:09.104138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:09.104559 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:09.104258    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:09.104559 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:09.104321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:09.104559 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:09.104432    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:10.104954 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:10.104918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:10.105375 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:10.105054    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:11.104780 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:11.104751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:11.105004 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:11.104753    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:11.105004 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.104886    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:11.105004 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.104948    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:11.696378 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:11.696344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:11.696572 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.696497    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:28:11.696572 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.696551    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs podName:fe629083-72c1-416f-a8ab-38fe211e7598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:27.696537044 +0000 UTC m=+34.122274453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs") pod "network-metrics-daemon-qfbg6" (UID: "fe629083-72c1-416f-a8ab-38fe211e7598") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:28:11.797301 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:11.797265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:11.797492 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.797390    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:28:11.797492 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.797468    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret podName:6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:27.797446424 +0000 UTC m=+34.223183844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret") pod "global-pull-secret-syncer-8tb9v" (UID: "6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:28:11.898213 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:11.898180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:11.898377 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.898357    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:28:11.898420 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.898378    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:28:11.898420 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.898388    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6ds6k for pod openshift-network-diagnostics/network-check-target-c7jb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:28:11.898512 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:11.898441    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k podName:8abd9fae-29f2-42aa-a064-3c1928c21286 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:27.898423741 +0000 UTC m=+34.324161152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ds6k" (UniqueName: "kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k") pod "network-check-target-c7jb9" (UID: "8abd9fae-29f2-42aa-a064-3c1928c21286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:28:12.104292 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:12.104210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:12.104460 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:12.104320    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:13.104367 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:13.104331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:13.104834 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:13.104377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:13.104834 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:13.104462    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:13.104834 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:13.104596    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:14.105667 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:14.105633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:14.106037 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:14.105743    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:15.105212 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.105027    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:15.105441 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.105072    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:15.105441 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:15.105308    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:15.105441 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:15.105375    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:15.272160 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.272075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp" event={"ID":"7dc378d9-85fc-47e5-93bd-5d7a31b5257d","Type":"ContainerStarted","Data":"f14cf6f0539fbf50ab1e624d8a1a0caa339c680a934f978653a3d27bc06c5c13"}
Sep 10 14:28:15.275235 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.275209    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:28:15.275612 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.275591    2566 generic.go:356] "Generic (PLEG): container finished" podID="055abf0a-a771-42fa-aa0a-be5cd2b73692" containerID="a7b54f1755448c39151aea3fccb1cf05ca604cd5b4f2e41563dc6befd3ceb438" exitCode=1
Sep 10 14:28:15.275719 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.275626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"5560e174f49c7be4116eac776a000f8a9b8723128279a0099b4346065759794b"}
Sep 10 14:28:15.275787 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.275732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"e36f9f0e9f8e319417c6f9c136642d4c565b83e7bce7688258798621e1e5f781"}
Sep 10 14:28:15.275837 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.275784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"dc8dc417a638d464802f9764dc2fa847fefb2fce0e5529daf7c47ffd9d778c30"}
Sep 10 14:28:15.275884 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.275834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"9502bbac6e8b6c8eb37e23326714f8805cb3300cd17b13062b74530f5fbd64c8"}
Sep 10 14:28:15.275931 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.275880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerDied","Data":"a7b54f1755448c39151aea3fccb1cf05ca604cd5b4f2e41563dc6befd3ceb438"}
Sep 10 14:28:15.275985 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.275931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"e05c91f672620430899f30f5185f40c0c5fa07bd0a07bdb5b0c13ed4540171e9"}
Sep 10 14:28:15.277284 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.277261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jr8gf" event={"ID":"3d97cd1e-f9fb-4f92-ab4c-2068152f5d01","Type":"ContainerStarted","Data":"8b727decbfcd4a09eb7676629b50649fb1a14759b51b3ff524bd5a13121c4029"}
Sep 10 14:28:15.278908 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.278883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tc6th" event={"ID":"65fab78a-e493-4fc4-b491-bc6bd32e73da","Type":"ContainerStarted","Data":"c0f6285425c026c4310205fe68d587a0c80c2c4e0cc618084ef9b15f04db5a1c"}
Sep 10 14:28:15.280498 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.280477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxhmb" event={"ID":"47492653-52e9-41d3-94e3-616fef6d8780","Type":"ContainerStarted","Data":"3dd8ae98b2ce30d8468689c9eadc8083896d03b383a6333ca71e46ced8cf7937"}
Sep 10 14:28:15.282162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.282132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fvv6t" event={"ID":"532db11a-6291-4687-8d65-e71384f16400","Type":"ContainerStarted","Data":"df435df13cc6602db85b78023416a930db39dbf988e14645769c11559966f230"}
Sep 10 14:28:15.283723 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.283686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5n69" event={"ID":"352316f0-970c-43ad-befe-ffeaaad7e582","Type":"ContainerStarted","Data":"1febb3216b84dcb9ad36a4694db8bba9a62db390adb1ec9ac1fb4460b50e27ee"}
Sep 10 14:28:15.285273 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.285251    2566 generic.go:356] "Generic (PLEG): container finished" podID="538b5488-3d52-4ae3-8ffc-5e3be2ac32de" containerID="a352c8b93dcbcf7703c408461eb75c83154105e99a6abbcf03abfe2ae5a7b7e1" exitCode=0
Sep 10 14:28:15.285363 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.285280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" event={"ID":"538b5488-3d52-4ae3-8ffc-5e3be2ac32de","Type":"ContainerDied","Data":"a352c8b93dcbcf7703c408461eb75c83154105e99a6abbcf03abfe2ae5a7b7e1"}
Sep 10 14:28:15.293205 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.293159    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jr8gf" podStartSLOduration=3.750040897 podStartE2EDuration="21.293148641s" totalImagesPullingTime="17.543107744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:15.292878463 +0000 UTC m=+21.718615884" watchObservedRunningTime="2026-09-10 14:28:15.293148641 +0000 UTC m=+21.718886059"
Sep 10 14:28:15.293539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.293495    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-162.ec2.internal" podStartSLOduration=20.293487663 podStartE2EDuration="20.293487663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:27:59.262851048 +0000 UTC m=+5.688588468" watchObservedRunningTime="2026-09-10 14:28:15.293487663 +0000 UTC m=+21.719225093"
Sep 10 14:28:15.307846 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.307601    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sxhmb" podStartSLOduration=8.240274376 podStartE2EDuration="21.307583373s" totalImagesPullingTime="13.067308997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:15.307145755 +0000 UTC m=+21.732883156" watchObservedRunningTime="2026-09-10 14:28:15.307583373 +0000 UTC m=+21.733320793"
Sep 10 14:28:15.358021 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.357955    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tc6th" podStartSLOduration=4.165791117 podStartE2EDuration="21.357933902s" totalImagesPullingTime="17.192142785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:15.341770226 +0000 UTC m=+21.767507635" watchObservedRunningTime="2026-09-10 14:28:15.357933902 +0000 UTC m=+21.783671322"
Sep 10 14:28:15.358177 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.358100    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fvv6t" podStartSLOduration=4.157634825 podStartE2EDuration="21.358094144s" totalImagesPullingTime="17.200459319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:15.357535994 +0000 UTC m=+21.783273413" watchObservedRunningTime="2026-09-10 14:28:15.358094144 +0000 UTC m=+21.783831563"
Sep 10 14:28:15.371281 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.371229    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v5n69" podStartSLOduration=8.274402428 podStartE2EDuration="21.371213774s" totalImagesPullingTime="13.096811346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:15.37082581 +0000 UTC m=+21.796563228" watchObservedRunningTime="2026-09-10 14:28:15.371213774 +0000 UTC m=+21.796951193"
Sep 10 14:28:15.440705 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.440474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:28:15.441286 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.441267    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:28:15.670779 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:15.670756    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 14:28:16.105073 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.105038    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:16.105412 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:16.105180    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:16.169396 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.169287    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T14:28:15.670774996Z","UUID":"76e33fd6-aa04-42d0-b64c-2952cbb6a4fd","Handler":null,"Name":"","Endpoint":""}
Sep 10 14:28:16.171000 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.170970    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 14:28:16.171120 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.171011    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 14:28:16.290499 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.290461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp" event={"ID":"7dc378d9-85fc-47e5-93bd-5d7a31b5257d","Type":"ContainerStarted","Data":"44ea3df51fe5ead6353516f2f0daba97133ddf14282e55d84302db32bfe07825"}
Sep 10 14:28:16.292357 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.292328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6254" event={"ID":"07145279-d28b-4f9a-9a9b-d7b763413fd1","Type":"ContainerStarted","Data":"490c9b5c536c3e97cdd76aad0cd386fcf0b335c28477b1f31d5e2e1d792d5353"}
Sep 10 14:28:16.293388 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.293368    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:28:16.293848 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.293826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sxhmb"
Sep 10 14:28:16.308761 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:16.308718    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k6254" podStartSLOduration=5.085488168 podStartE2EDuration="22.308705697s" totalImagesPullingTime="17.223217529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:16.307857472 +0000 UTC m=+22.733594892" watchObservedRunningTime="2026-09-10 14:28:16.308705697 +0000 UTC m=+22.734443107"
Sep 10 14:28:17.104800 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:17.104772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:17.104800 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:17.104790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:17.104943 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:17.104924    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:17.105020 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:17.104994    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:17.297278 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:17.297236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp" event={"ID":"7dc378d9-85fc-47e5-93bd-5d7a31b5257d","Type":"ContainerStarted","Data":"ee7559a12f31ce3a5be6ca1b3dcb0c7cc48cc8f3513c7b0bb3ce8ac4af3ba91a"}
Sep 10 14:28:17.300707 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:17.300671    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:28:17.301106 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:17.301065    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"0270055aefea2f85ef6749cda8586cb5509ea7c950778c62636377da7d69f336"}
Sep 10 14:28:17.314005 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:17.313951    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5fqfp" podStartSLOduration=4.230951888 podStartE2EDuration="23.313937358s" totalImagesPullingTime="19.08298547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:17.313757344 +0000 UTC m=+23.739494762" watchObservedRunningTime="2026-09-10 14:28:17.313937358 +0000 UTC m=+23.739674778"
Sep 10 14:28:18.104767 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:18.104734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:18.105016 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:18.104849    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:19.104789 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:19.104755    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:19.105268 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:19.104805    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:19.105268 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:19.104878    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:19.105268 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:19.105014    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:19.309336 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:19.309133    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:28:19.309703 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:19.309667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"ecd1f7af268ba815c1f2f8ceb81fae891bcfcbc1bf2dc49a15323e80e267e001"}
Sep 10 14:28:19.311908 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:19.311473    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:28:19.311908 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:19.311599    2566 scope.go:117] "RemoveContainer" containerID="a7b54f1755448c39151aea3fccb1cf05ca604cd5b4f2e41563dc6befd3ceb438"
Sep 10 14:28:19.330124 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:19.330106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:28:20.105282 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.105251    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:20.105937 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:20.105350    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:20.315602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.315571    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:28:20.316020 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.316000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" event={"ID":"055abf0a-a771-42fa-aa0a-be5cd2b73692","Type":"ContainerStarted","Data":"8f4843413a7d98e5fcb92c6e9d95d967f8b4f12125edc60e4a209c856a79d35c"}
Sep 10 14:28:20.316292 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.316275    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:28:20.316381 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.316366    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:28:20.317947 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.317919    2566 generic.go:356] "Generic (PLEG): container finished" podID="538b5488-3d52-4ae3-8ffc-5e3be2ac32de" containerID="dd1363e00ddecdff3af177c070369a7d6f77de7fddd7d9265a96c58b396c9c24" exitCode=0
Sep 10 14:28:20.318083 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.317956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" event={"ID":"538b5488-3d52-4ae3-8ffc-5e3be2ac32de","Type":"ContainerDied","Data":"dd1363e00ddecdff3af177c070369a7d6f77de7fddd7d9265a96c58b396c9c24"}
Sep 10 14:28:20.335100 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.335079    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:28:20.343065 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:20.343028    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr" podStartSLOduration=9.133211456 podStartE2EDuration="26.343016237s" totalImagesPullingTime="17.209804781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:20.342029321 +0000 UTC m=+26.767766739" watchObservedRunningTime="2026-09-10 14:28:20.343016237 +0000 UTC m=+26.768753655"
Sep 10 14:28:21.105221 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.105039    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:21.105418 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.105079    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:21.105418 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:21.105308    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:21.105793 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:21.105468    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:21.115154 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.115126    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c7jb9"]
Sep 10 14:28:21.121288 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.121243    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8tb9v"]
Sep 10 14:28:21.121424 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.121355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:21.121496 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:21.121473    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:21.122005 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.121987    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qfbg6"]
Sep 10 14:28:21.322411 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.322380    2566 generic.go:356] "Generic (PLEG): container finished" podID="538b5488-3d52-4ae3-8ffc-5e3be2ac32de" containerID="2bb8c30393a99e209decbc19172c1a5d506d8e49049916343c5dbd63abda19fc" exitCode=0
Sep 10 14:28:21.322535 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.322467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" event={"ID":"538b5488-3d52-4ae3-8ffc-5e3be2ac32de","Type":"ContainerDied","Data":"2bb8c30393a99e209decbc19172c1a5d506d8e49049916343c5dbd63abda19fc"}
Sep 10 14:28:21.322535 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.322515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:21.322636 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:21.322602    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:21.322971 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:21.322953    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:21.323081 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:21.323019    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:22.326848 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:22.326764    2566 generic.go:356] "Generic (PLEG): container finished" podID="538b5488-3d52-4ae3-8ffc-5e3be2ac32de" containerID="85b8762d7c2c6d979a56c8f6b909cf8ef5c7f1fbb38cb578f9f78709ee99c7b8" exitCode=0
Sep 10 14:28:22.327361 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:22.326848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" event={"ID":"538b5488-3d52-4ae3-8ffc-5e3be2ac32de","Type":"ContainerDied","Data":"85b8762d7c2c6d979a56c8f6b909cf8ef5c7f1fbb38cb578f9f78709ee99c7b8"}
Sep 10 14:28:23.104984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:23.104848    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:23.104984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:23.104870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:23.104984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:23.104867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:23.105233 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:23.105007    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:23.105233 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:23.105057    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:23.105233 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:23.105144    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:25.104469 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:25.104434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:25.104950 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:25.104599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:25.104950 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:25.104712    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:25.104950 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:25.104570    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:25.104950 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:25.104793    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:25.104950 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:25.104885    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:27.105101 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.104899    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:27.105544 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.104955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:27.105544 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.105176    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tb9v" podUID="6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4"
Sep 10 14:28:27.105544 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.104965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:27.105544 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.105244    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c7jb9" podUID="8abd9fae-29f2-42aa-a064-3c1928c21286"
Sep 10 14:28:27.105544 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.105368    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfbg6" podUID="fe629083-72c1-416f-a8ab-38fe211e7598"
Sep 10 14:28:27.317163 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.317126    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeReady"
Sep 10 14:28:27.317318 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.317248    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 14:28:27.341045 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.341013    2566 generic.go:356] "Generic (PLEG): container finished" podID="538b5488-3d52-4ae3-8ffc-5e3be2ac32de" containerID="5c1272800cce8a224deaeef42e0ae2cabd09323c036748ae790ddf55af2be76b" exitCode=0
Sep 10 14:28:27.341045 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.341048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" event={"ID":"538b5488-3d52-4ae3-8ffc-5e3be2ac32de","Type":"ContainerDied","Data":"5c1272800cce8a224deaeef42e0ae2cabd09323c036748ae790ddf55af2be76b"}
Sep 10 14:28:27.361136 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.361070    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-98m8q"]
Sep 10 14:28:27.376766 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.376738    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p26lr"]
Sep 10 14:28:27.377094 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.377071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.379291 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.379117    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 14:28:27.379397 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.379363    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lsf8q\""
Sep 10 14:28:27.379449 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.379406    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 14:28:27.390646 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.390623    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98m8q"]
Sep 10 14:28:27.390784 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.390666    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p26lr"]
Sep 10 14:28:27.390784 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.390752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p26lr"
Sep 10 14:28:27.393881 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.393858    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 14:28:27.393881 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.393878    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 14:28:27.394120 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.394104    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z67dn\""
Sep 10 14:28:27.394243 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.394227    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 14:28:27.519097 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.519064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd04e0bc-77f9-44c0-87a9-d8123246c487-tmp-dir\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.519223 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.519107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhts\" (UniqueName: \"kubernetes.io/projected/cd04e0bc-77f9-44c0-87a9-d8123246c487-kube-api-access-7mhts\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.519223 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.519135    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd04e0bc-77f9-44c0-87a9-d8123246c487-config-volume\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.519223 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.519159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.519223 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.519177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgfj\" (UniqueName: \"kubernetes.io/projected/312e6872-7076-44d1-8bbc-0c2b9b4161d5-kube-api-access-5pgfj\") pod \"ingress-canary-p26lr\" (UID: \"312e6872-7076-44d1-8bbc-0c2b9b4161d5\") " pod="openshift-ingress-canary/ingress-canary-p26lr"
Sep 10 14:28:27.519345 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.519271    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312e6872-7076-44d1-8bbc-0c2b9b4161d5-cert\") pod \"ingress-canary-p26lr\" (UID: \"312e6872-7076-44d1-8bbc-0c2b9b4161d5\") " pod="openshift-ingress-canary/ingress-canary-p26lr"
Sep 10 14:28:27.620491 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.620400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312e6872-7076-44d1-8bbc-0c2b9b4161d5-cert\") pod \"ingress-canary-p26lr\" (UID: \"312e6872-7076-44d1-8bbc-0c2b9b4161d5\") " pod="openshift-ingress-canary/ingress-canary-p26lr"
Sep 10 14:28:27.620491 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.620444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd04e0bc-77f9-44c0-87a9-d8123246c487-tmp-dir\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.620491 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.620467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhts\" (UniqueName: \"kubernetes.io/projected/cd04e0bc-77f9-44c0-87a9-d8123246c487-kube-api-access-7mhts\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.620491 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.620486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd04e0bc-77f9-44c0-87a9-d8123246c487-config-volume\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.620874 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.620507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.620874 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.620533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgfj\" (UniqueName: \"kubernetes.io/projected/312e6872-7076-44d1-8bbc-0c2b9b4161d5-kube-api-access-5pgfj\") pod \"ingress-canary-p26lr\" (UID: \"312e6872-7076-44d1-8bbc-0c2b9b4161d5\") " pod="openshift-ingress-canary/ingress-canary-p26lr"
Sep 10 14:28:27.620874 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.620660    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:28:27.620874 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.620748    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls podName:cd04e0bc-77f9-44c0-87a9-d8123246c487 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:28.120725513 +0000 UTC m=+34.546462929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls") pod "dns-default-98m8q" (UID: "cd04e0bc-77f9-44c0-87a9-d8123246c487") : secret "dns-default-metrics-tls" not found
Sep 10 14:28:27.621061 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.620912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd04e0bc-77f9-44c0-87a9-d8123246c487-tmp-dir\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.621109 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.621087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd04e0bc-77f9-44c0-87a9-d8123246c487-config-volume\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.624534 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.624510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312e6872-7076-44d1-8bbc-0c2b9b4161d5-cert\") pod \"ingress-canary-p26lr\" (UID: \"312e6872-7076-44d1-8bbc-0c2b9b4161d5\") " pod="openshift-ingress-canary/ingress-canary-p26lr"
Sep 10 14:28:27.631742 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.631717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgfj\" (UniqueName: \"kubernetes.io/projected/312e6872-7076-44d1-8bbc-0c2b9b4161d5-kube-api-access-5pgfj\") pod \"ingress-canary-p26lr\" (UID: \"312e6872-7076-44d1-8bbc-0c2b9b4161d5\") " pod="openshift-ingress-canary/ingress-canary-p26lr"
Sep 10 14:28:27.632251 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.632235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhts\" (UniqueName: \"kubernetes.io/projected/cd04e0bc-77f9-44c0-87a9-d8123246c487-kube-api-access-7mhts\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:27.700450 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.700410    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p26lr"
Sep 10 14:28:27.721112 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.720935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:27.721112 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.721075    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:28:27.721228 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.721142    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs podName:fe629083-72c1-416f-a8ab-38fe211e7598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:59.721124264 +0000 UTC m=+66.146861679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs") pod "network-metrics-daemon-qfbg6" (UID: "fe629083-72c1-416f-a8ab-38fe211e7598") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:28:27.822221 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.822180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:27.822409 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.822335    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:28:27.822409 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.822403    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret podName:6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:59.822389628 +0000 UTC m=+66.248127025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret") pod "global-pull-secret-syncer-8tb9v" (UID: "6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:28:27.923330 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.923238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:27.923465 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.923403    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:28:27.923465 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.923423    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:28:27.923465 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.923432    2566 projected.go:196] Error preparing data for projected volume kube-api-access-6ds6k for pod openshift-network-diagnostics/network-check-target-c7jb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:28:27.923614 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:27.923485    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k podName:8abd9fae-29f2-42aa-a064-3c1928c21286 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:59.92346967 +0000 UTC m=+66.349207090 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ds6k" (UniqueName: "kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k") pod "network-check-target-c7jb9" (UID: "8abd9fae-29f2-42aa-a064-3c1928c21286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:28:27.998054 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:27.998024    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p26lr"]
Sep 10 14:28:28.001774 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:28:28.001749    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312e6872_7076_44d1_8bbc_0c2b9b4161d5.slice/crio-fbfdc0fb6179864c97bdedd289f7060fa11f77505048be09fa3fd7368fe63590 WatchSource:0}: Error finding container fbfdc0fb6179864c97bdedd289f7060fa11f77505048be09fa3fd7368fe63590: Status 404 returned error can't find the container with id fbfdc0fb6179864c97bdedd289f7060fa11f77505048be09fa3fd7368fe63590
Sep 10 14:28:28.125847 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:28.125811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:28.126505 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:28.125966    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:28:28.126505 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:28.126035    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls podName:cd04e0bc-77f9-44c0-87a9-d8123246c487 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:29.126018935 +0000 UTC m=+35.551756336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls") pod "dns-default-98m8q" (UID: "cd04e0bc-77f9-44c0-87a9-d8123246c487") : secret "dns-default-metrics-tls" not found
Sep 10 14:28:28.346232 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:28.346143    2566 generic.go:356] "Generic (PLEG): container finished" podID="538b5488-3d52-4ae3-8ffc-5e3be2ac32de" containerID="68ffa1040d84c71dc0c9aec0cb866ae2f1872f4acf0dbb1dfa1d725ad3ab933f" exitCode=0
Sep 10 14:28:28.346386 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:28.346222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" event={"ID":"538b5488-3d52-4ae3-8ffc-5e3be2ac32de","Type":"ContainerDied","Data":"68ffa1040d84c71dc0c9aec0cb866ae2f1872f4acf0dbb1dfa1d725ad3ab933f"}
Sep 10 14:28:28.347304 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:28.347285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p26lr" event={"ID":"312e6872-7076-44d1-8bbc-0c2b9b4161d5","Type":"ContainerStarted","Data":"fbfdc0fb6179864c97bdedd289f7060fa11f77505048be09fa3fd7368fe63590"}
Sep 10 14:28:29.104611 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.104385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:29.104817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.104384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:29.104817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.104384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:29.106942 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.106921    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:28:29.107898 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.107624    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:28:29.107898 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.107645    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pptck\""
Sep 10 14:28:29.107898 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.107733    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:28:29.107898 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.107746    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bf25v\""
Sep 10 14:28:29.107898 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.107847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:28:29.131975 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.131954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:29.132259 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:29.132081    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:28:29.132259 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:29.132130    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls podName:cd04e0bc-77f9-44c0-87a9-d8123246c487 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:31.132117915 +0000 UTC m=+37.557855316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls") pod "dns-default-98m8q" (UID: "cd04e0bc-77f9-44c0-87a9-d8123246c487") : secret "dns-default-metrics-tls" not found
Sep 10 14:28:29.353217 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.353181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" event={"ID":"538b5488-3d52-4ae3-8ffc-5e3be2ac32de","Type":"ContainerStarted","Data":"2500fdaf7c362e3c596f33ea1972e75c67ad5d4df0de54ce17485adb47497b2f"}
Sep 10 14:28:29.372935 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.372841    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bpg6r" podStartSLOduration=8.613079052 podStartE2EDuration="35.372823595s" totalImagesPullingTime="26.759744543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:29.372294885 +0000 UTC m=+35.798032302" watchObservedRunningTime="2026-09-10 14:28:29.372823595 +0000 UTC m=+35.798561013"
Sep 10 14:28:29.744315 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.744225    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"]
Sep 10 14:28:29.774933 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.774905    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"]
Sep 10 14:28:29.775087 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.775007    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:29.777448 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.777415    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 14:28:29.777448 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.777432    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 14:28:29.777757 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.777711    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:28:29.777818 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.777788    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 14:28:29.777935 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.777913    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5vzzs\""
Sep 10 14:28:29.844503 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.844469    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"]
Sep 10 14:28:29.861239 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.861202    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"]
Sep 10 14:28:29.861239 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.861246    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm"]
Sep 10 14:28:29.861453 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.861288    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:29.863533 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.863507    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 14:28:29.863658 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.863515    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 14:28:29.863658 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.863579    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qsd67\""
Sep 10 14:28:29.863911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.863881    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:28:29.863988 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.863932    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 14:28:29.882907 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.882881    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bfwnd"]
Sep 10 14:28:29.883367 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.883344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm"
Sep 10 14:28:29.885538 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.885475    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wqb5l\""
Sep 10 14:28:29.885538 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.885481    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 14:28:29.885717 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.885591    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:28:29.909214 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.909163    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"]
Sep 10 14:28:29.909411 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.909301    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:29.911786 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.911719    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 14:28:29.911913 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.911895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jpkgl\""
Sep 10 14:28:29.912010 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.911993    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 14:28:29.912088 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.912076    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 14:28:29.912292 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.912263    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:28:29.919506 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.919477    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 14:28:29.923124 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.923102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm"]
Sep 10 14:28:29.923242 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.923135    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bfwnd"]
Sep 10 14:28:29.923242 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.923160    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"]
Sep 10 14:28:29.923242 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.923225    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:29.925331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.925308    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mztbj\""
Sep 10 14:28:29.925552 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.925320    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 14:28:29.925680 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.925390    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:28:29.925680 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.925432    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 14:28:29.940569 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.940546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96ede326-292a-4951-9185-329b2e579f71-config\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:29.940710 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.940594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kl2\" (UniqueName: \"kubernetes.io/projected/96ede326-292a-4951-9185-329b2e579f71-kube-api-access-p8kl2\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:29.940710 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.940627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmbnb\" (UniqueName: \"kubernetes.io/projected/7db503b0-a2c2-415b-9481-6b65ad9b7d23-kube-api-access-zmbnb\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:29.940710 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.940658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7db503b0-a2c2-415b-9481-6b65ad9b7d23-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:29.940710 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.940682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7db503b0-a2c2-415b-9481-6b65ad9b7d23-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:29.940924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.940716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96ede326-292a-4951-9185-329b2e579f71-serving-cert\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:29.954188 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.954162    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"]
Sep 10 14:28:29.965540 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.965512    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6677bb96-ft5cx"]
Sep 10 14:28:29.965666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.965640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:29.970381 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.970359    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pxrpp\""
Sep 10 14:28:29.971419 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.970751    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 14:28:29.971931 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.971760    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 14:28:29.971931 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.971818    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 14:28:29.971931 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.971772    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 14:28:29.975286 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.975252    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"]
Sep 10 14:28:29.975392 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.975354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:29.977292 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.977273    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2kwcf\""
Sep 10 14:28:29.977642 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.977625    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 14:28:29.977936 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.977918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 14:28:29.978029 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.977965    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 14:28:29.982829 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.982811    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 14:28:29.989721 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:29.989686    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6677bb96-ft5cx"]
Sep 10 14:28:30.041754 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.041668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:30.041754 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.041729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.041962 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.041764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96ede326-292a-4951-9185-329b2e579f71-config\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:30.041962 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.041784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zz2r\" (UniqueName: \"kubernetes.io/projected/ee67ac4c-3579-4d26-a029-9f782d31589f-kube-api-access-9zz2r\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.041962 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.041925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kl2\" (UniqueName: \"kubernetes.io/projected/96ede326-292a-4951-9185-329b2e579f71-kube-api-access-p8kl2\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:30.042115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.041991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-serving-cert\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.042115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmbnb\" (UniqueName: \"kubernetes.io/projected/7db503b0-a2c2-415b-9481-6b65ad9b7d23-kube-api-access-zmbnb\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:30.042115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-trusted-ca\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.042115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-config\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.042307 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7db503b0-a2c2-415b-9481-6b65ad9b7d23-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:30.042307 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7db503b0-a2c2-415b-9481-6b65ad9b7d23-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:30.042307 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96ede326-292a-4951-9185-329b2e579f71-serving-cert\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:30.042307 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ee67ac4c-3579-4d26-a029-9f782d31589f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.042307 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kml9m\" (UniqueName: \"kubernetes.io/projected/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-kube-api-access-kml9m\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.042307 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5d6\" (UniqueName: \"kubernetes.io/projected/8113e71f-67b8-4cbb-90b1-7e85d15ad598-kube-api-access-kr5d6\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:30.042307 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrq7p\" (UniqueName: \"kubernetes.io/projected/0538b272-5201-49b4-bff0-69c7146973ab-kube-api-access-zrq7p\") pod \"volume-data-source-validator-695c895bd6-djwzm\" (UID: \"0538b272-5201-49b4-bff0-69c7146973ab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm"
Sep 10 14:28:30.042644 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96ede326-292a-4951-9185-329b2e579f71-config\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:30.042881 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.042854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7db503b0-a2c2-415b-9481-6b65ad9b7d23-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:30.045486 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.045465    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7db503b0-a2c2-415b-9481-6b65ad9b7d23-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:30.045581 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.045466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96ede326-292a-4951-9185-329b2e579f71-serving-cert\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:30.049795 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.049745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kl2\" (UniqueName: \"kubernetes.io/projected/96ede326-292a-4951-9185-329b2e579f71-kube-api-access-p8kl2\") pod \"service-ca-operator-859f94c4b-tt6bd\" (UID: \"96ede326-292a-4951-9185-329b2e579f71\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:30.049917 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.049822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmbnb\" (UniqueName: \"kubernetes.io/projected/7db503b0-a2c2-415b-9481-6b65ad9b7d23-kube-api-access-zmbnb\") pod \"kube-storage-version-migrator-operator-8d7dd4675-f4rj9\" (UID: \"7db503b0-a2c2-415b-9481-6b65ad9b7d23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:30.084809 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.084776    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"
Sep 10 14:28:30.143275 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-trusted-ca\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-config\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/215e4021-d8eb-421d-8095-6e3f9c87e0bb-ca-trust-extracted\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74689\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-kube-api-access-74689\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-image-registry-private-configuration\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-trusted-ca\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-installation-pull-secrets\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-serving-cert\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-certificates\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ee67ac4c-3579-4d26-a029-9f782d31589f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kml9m\" (UniqueName: \"kubernetes.io/projected/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-kube-api-access-kml9m\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.143664    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:30.143683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5d6\" (UniqueName: \"kubernetes.io/projected/8113e71f-67b8-4cbb-90b1-7e85d15ad598-kube-api-access-kr5d6\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:30.144234 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrq7p\" (UniqueName: \"kubernetes.io/projected/0538b272-5201-49b4-bff0-69c7146973ab-kube-api-access-zrq7p\") pod \"volume-data-source-validator-695c895bd6-djwzm\" (UID: \"0538b272-5201-49b4-bff0-69c7146973ab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm"
Sep 10 14:28:30.144234 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-bound-sa-token\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.144234 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.143767    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls podName:ee67ac4c-3579-4d26-a029-9f782d31589f nodeName:}" failed. No retries permitted until 2026-09-10 14:28:30.643746338 +0000 UTC m=+37.069483738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lnvg9" (UID: "ee67ac4c-3579-4d26-a029-9f782d31589f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:30.144234 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:30.144234 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.143835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zz2r\" (UniqueName: \"kubernetes.io/projected/ee67ac4c-3579-4d26-a029-9f782d31589f-kube-api-access-9zz2r\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.144234 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.144026    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:28:30.144234 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.144091    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls podName:8113e71f-67b8-4cbb-90b1-7e85d15ad598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:30.644074238 +0000 UTC m=+37.069811650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kj6xm" (UID: "8113e71f-67b8-4cbb-90b1-7e85d15ad598") : secret "samples-operator-tls" not found
Sep 10 14:28:30.144234 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.144112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-config\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.144513 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.144339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-trusted-ca\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.144804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.144786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ee67ac4c-3579-4d26-a029-9f782d31589f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.146544 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.146516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-serving-cert\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.155298 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.155271    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zz2r\" (UniqueName: \"kubernetes.io/projected/ee67ac4c-3579-4d26-a029-9f782d31589f-kube-api-access-9zz2r\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.157930 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.157905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrq7p\" (UniqueName: \"kubernetes.io/projected/0538b272-5201-49b4-bff0-69c7146973ab-kube-api-access-zrq7p\") pod \"volume-data-source-validator-695c895bd6-djwzm\" (UID: \"0538b272-5201-49b4-bff0-69c7146973ab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm"
Sep 10 14:28:30.162621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.162598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5d6\" (UniqueName: \"kubernetes.io/projected/8113e71f-67b8-4cbb-90b1-7e85d15ad598-kube-api-access-kr5d6\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:30.164807 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.164788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kml9m\" (UniqueName: \"kubernetes.io/projected/2d4f3935-7fa7-4cd5-aa0b-a5178e683e41-kube-api-access-kml9m\") pod \"console-operator-85b454874b-bfwnd\" (UID: \"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41\") " pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.171622 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.171601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"
Sep 10 14:28:30.193782 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.193762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm"
Sep 10 14:28:30.222059 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.221885    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:30.245476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.244611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-certificates\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.245476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.244672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-bound-sa-token\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.245476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.244895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/215e4021-d8eb-421d-8095-6e3f9c87e0bb-ca-trust-extracted\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.245476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.244927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.245476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.244954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74689\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-kube-api-access-74689\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.245476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.244987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-image-registry-private-configuration\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.245476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.245016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-trusted-ca\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.245476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.245046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-installation-pull-secrets\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.246070 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.245575    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:28:30.246070 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.245594    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6677bb96-ft5cx: secret "image-registry-tls" not found
Sep 10 14:28:30.247919 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.247681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-trusted-ca\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.247919 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.247775    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls podName:215e4021-d8eb-421d-8095-6e3f9c87e0bb nodeName:}" failed. No retries permitted until 2026-09-10 14:28:30.745652148 +0000 UTC m=+37.171389568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls") pod "image-registry-6677bb96-ft5cx" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb") : secret "image-registry-tls" not found
Sep 10 14:28:30.249638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.248759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-certificates\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.249638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.249185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/215e4021-d8eb-421d-8095-6e3f9c87e0bb-ca-trust-extracted\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.251797 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.251438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-installation-pull-secrets\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.253720 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.253659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-image-registry-private-configuration\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.256400 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.256379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-bound-sa-token\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.257204 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.257183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74689\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-kube-api-access-74689\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.396583 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.396384    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9"]
Sep 10 14:28:30.432995 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:28:30.432950    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db503b0_a2c2_415b_9481_6b65ad9b7d23.slice/crio-e51f997f7b7877cbd6ff31ebfca284b7981a087df12d1811655dea88fc33f463 WatchSource:0}: Error finding container e51f997f7b7877cbd6ff31ebfca284b7981a087df12d1811655dea88fc33f463: Status 404 returned error can't find the container with id e51f997f7b7877cbd6ff31ebfca284b7981a087df12d1811655dea88fc33f463
Sep 10 14:28:30.451454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.451424    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd"]
Sep 10 14:28:30.455287 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:28:30.455255    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ede326_292a_4951_9185_329b2e579f71.slice/crio-b24a45091a86fba21ea7f00686c3318e9a5505215827690fb1eef64d92965521 WatchSource:0}: Error finding container b24a45091a86fba21ea7f00686c3318e9a5505215827690fb1eef64d92965521: Status 404 returned error can't find the container with id b24a45091a86fba21ea7f00686c3318e9a5505215827690fb1eef64d92965521
Sep 10 14:28:30.482931 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.482899    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm"]
Sep 10 14:28:30.485597 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:28:30.485575    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0538b272_5201_49b4_bff0_69c7146973ab.slice/crio-72dd35025e71124afca694015b90a675469a1a60548ac4ea3ae7dffdf2224e3a WatchSource:0}: Error finding container 72dd35025e71124afca694015b90a675469a1a60548ac4ea3ae7dffdf2224e3a: Status 404 returned error can't find the container with id 72dd35025e71124afca694015b90a675469a1a60548ac4ea3ae7dffdf2224e3a
Sep 10 14:28:30.514661 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.514635    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bfwnd"]
Sep 10 14:28:30.517178 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:28:30.517150    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4f3935_7fa7_4cd5_aa0b_a5178e683e41.slice/crio-23b464c242d8c033673b2aba6669888b2d88b10deda0b792151239039832285b WatchSource:0}: Error finding container 23b464c242d8c033673b2aba6669888b2d88b10deda0b792151239039832285b: Status 404 returned error can't find the container with id 23b464c242d8c033673b2aba6669888b2d88b10deda0b792151239039832285b
Sep 10 14:28:30.648552 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.648506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:30.648771 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.648576    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:30.648771 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.648669    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:28:30.648771 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.648674    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:30.648771 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.648741    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls podName:8113e71f-67b8-4cbb-90b1-7e85d15ad598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:31.648726699 +0000 UTC m=+38.074464096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kj6xm" (UID: "8113e71f-67b8-4cbb-90b1-7e85d15ad598") : secret "samples-operator-tls" not found
Sep 10 14:28:30.648771 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.648754    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls podName:ee67ac4c-3579-4d26-a029-9f782d31589f nodeName:}" failed. No retries permitted until 2026-09-10 14:28:31.648748878 +0000 UTC m=+38.074486275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lnvg9" (UID: "ee67ac4c-3579-4d26-a029-9f782d31589f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:30.749223 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:30.749188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:30.749383 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.749349    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:28:30.749383 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.749369    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6677bb96-ft5cx: secret "image-registry-tls" not found
Sep 10 14:28:30.749455 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:30.749424    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls podName:215e4021-d8eb-421d-8095-6e3f9c87e0bb nodeName:}" failed. No retries permitted until 2026-09-10 14:28:31.749407628 +0000 UTC m=+38.175145026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls") pod "image-registry-6677bb96-ft5cx" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb") : secret "image-registry-tls" not found
Sep 10 14:28:31.153920 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.153884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:31.154477 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.154060    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:28:31.154477 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.154141    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls podName:cd04e0bc-77f9-44c0-87a9-d8123246c487 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:35.154119761 +0000 UTC m=+41.579857169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls") pod "dns-default-98m8q" (UID: "cd04e0bc-77f9-44c0-87a9-d8123246c487") : secret "dns-default-metrics-tls" not found
Sep 10 14:28:31.361045 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.361002    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bfwnd" event={"ID":"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41","Type":"ContainerStarted","Data":"23b464c242d8c033673b2aba6669888b2d88b10deda0b792151239039832285b"}
Sep 10 14:28:31.363629 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.363597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd" event={"ID":"96ede326-292a-4951-9185-329b2e579f71","Type":"ContainerStarted","Data":"b24a45091a86fba21ea7f00686c3318e9a5505215827690fb1eef64d92965521"}
Sep 10 14:28:31.365389 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.365363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9" event={"ID":"7db503b0-a2c2-415b-9481-6b65ad9b7d23","Type":"ContainerStarted","Data":"e51f997f7b7877cbd6ff31ebfca284b7981a087df12d1811655dea88fc33f463"}
Sep 10 14:28:31.368088 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.368063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p26lr" event={"ID":"312e6872-7076-44d1-8bbc-0c2b9b4161d5","Type":"ContainerStarted","Data":"6c33f2eb538c0dfce53263bcdba8a326a821e41451a2499777494971bb2ef829"}
Sep 10 14:28:31.371208 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.371187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm" event={"ID":"0538b272-5201-49b4-bff0-69c7146973ab","Type":"ContainerStarted","Data":"72dd35025e71124afca694015b90a675469a1a60548ac4ea3ae7dffdf2224e3a"}
Sep 10 14:28:31.384093 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.383449    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p26lr" podStartSLOduration=1.937550536 podStartE2EDuration="4.383430857s" totalImagesPullingTime="2.445880321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:31.383010816 +0000 UTC m=+37.808748237" watchObservedRunningTime="2026-09-10 14:28:31.383430857 +0000 UTC m=+37.809168278"
Sep 10 14:28:31.659657 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.658821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:31.659657 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.658942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:31.659657 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.659097    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:31.659657 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.659162    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls podName:ee67ac4c-3579-4d26-a029-9f782d31589f nodeName:}" failed. No retries permitted until 2026-09-10 14:28:33.659145453 +0000 UTC m=+40.084882857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lnvg9" (UID: "ee67ac4c-3579-4d26-a029-9f782d31589f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:31.659657 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.659575    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:28:31.659657 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.659623    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls podName:8113e71f-67b8-4cbb-90b1-7e85d15ad598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:33.659609204 +0000 UTC m=+40.085346604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kj6xm" (UID: "8113e71f-67b8-4cbb-90b1-7e85d15ad598") : secret "samples-operator-tls" not found
Sep 10 14:28:31.760771 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:31.760338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:31.760771 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.760659    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:28:31.760771 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.760679    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6677bb96-ft5cx: secret "image-registry-tls" not found
Sep 10 14:28:31.760771 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:31.760757    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls podName:215e4021-d8eb-421d-8095-6e3f9c87e0bb nodeName:}" failed. No retries permitted until 2026-09-10 14:28:33.760738452 +0000 UTC m=+40.186475873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls") pod "image-registry-6677bb96-ft5cx" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb") : secret "image-registry-tls" not found
Sep 10 14:28:33.675663 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:33.675583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:33.676122 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:33.675685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:33.676122 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:33.675750    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:28:33.676122 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:33.675820    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls podName:8113e71f-67b8-4cbb-90b1-7e85d15ad598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:37.675805334 +0000 UTC m=+44.101542736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kj6xm" (UID: "8113e71f-67b8-4cbb-90b1-7e85d15ad598") : secret "samples-operator-tls" not found
Sep 10 14:28:33.676122 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:33.675849    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:33.676122 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:33.675904    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls podName:ee67ac4c-3579-4d26-a029-9f782d31589f nodeName:}" failed. No retries permitted until 2026-09-10 14:28:37.675891149 +0000 UTC m=+44.101628547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lnvg9" (UID: "ee67ac4c-3579-4d26-a029-9f782d31589f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:33.776627 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:33.776559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:33.776816 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:33.776746    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:28:33.776816 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:33.776766    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6677bb96-ft5cx: secret "image-registry-tls" not found
Sep 10 14:28:33.776904 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:33.776823    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls podName:215e4021-d8eb-421d-8095-6e3f9c87e0bb nodeName:}" failed. No retries permitted until 2026-09-10 14:28:37.77680759 +0000 UTC m=+44.202544988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls") pod "image-registry-6677bb96-ft5cx" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb") : secret "image-registry-tls" not found
Sep 10 14:28:35.191920 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.191871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:35.192295 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:35.192029    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:28:35.192295 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:35.192114    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls podName:cd04e0bc-77f9-44c0-87a9-d8123246c487 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:43.192096941 +0000 UTC m=+49.617834352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls") pod "dns-default-98m8q" (UID: "cd04e0bc-77f9-44c0-87a9-d8123246c487") : secret "dns-default-metrics-tls" not found
Sep 10 14:28:35.382666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.382637    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/0.log"
Sep 10 14:28:35.382830 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.382676    2566 generic.go:356] "Generic (PLEG): container finished" podID="2d4f3935-7fa7-4cd5-aa0b-a5178e683e41" containerID="7cd1e6848ef36fcddd9a87da699d089df54bab50d3eab2bdec6fd505a7703d0d" exitCode=255
Sep 10 14:28:35.382830 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.382723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bfwnd" event={"ID":"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41","Type":"ContainerDied","Data":"7cd1e6848ef36fcddd9a87da699d089df54bab50d3eab2bdec6fd505a7703d0d"}
Sep 10 14:28:35.382990 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.382973    2566 scope.go:117] "RemoveContainer" containerID="7cd1e6848ef36fcddd9a87da699d089df54bab50d3eab2bdec6fd505a7703d0d"
Sep 10 14:28:35.384515 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.384485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd" event={"ID":"96ede326-292a-4951-9185-329b2e579f71","Type":"ContainerStarted","Data":"1328ef4160fc4acba1dd49c36d1586c750f8872dc3dc79859bce24603cbd2557"}
Sep 10 14:28:35.386599 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.386576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9" event={"ID":"7db503b0-a2c2-415b-9481-6b65ad9b7d23","Type":"ContainerStarted","Data":"295ea7d7ddc4b3d4774b3928c5d806781442f1f6b26360ec8119d3e807e1183e"}
Sep 10 14:28:35.388180 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.388156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm" event={"ID":"0538b272-5201-49b4-bff0-69c7146973ab","Type":"ContainerStarted","Data":"1a9118970f7f780730092f77fbe9e69b9a4d1a3067802e98ce9b9681e84d9caa"}
Sep 10 14:28:35.413553 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.413498    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-djwzm" podStartSLOduration=2.22126056 podStartE2EDuration="6.413476258s" totalImagesPullingTime="4.192215698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:35.412799824 +0000 UTC m=+41.838537270" watchObservedRunningTime="2026-09-10 14:28:35.413476258 +0000 UTC m=+41.839213682"
Sep 10 14:28:35.427168 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.427108    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9" podStartSLOduration=2.15767992 podStartE2EDuration="6.427094098s" totalImagesPullingTime="4.269414178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:35.42628953 +0000 UTC m=+41.852026949" watchObservedRunningTime="2026-09-10 14:28:35.427094098 +0000 UTC m=+41.852831514"
Sep 10 14:28:35.444375 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.444321    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd" podStartSLOduration=2.190182308 podStartE2EDuration="6.44430093s" totalImagesPullingTime="4.254118622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:35.444260258 +0000 UTC m=+41.869997689" watchObservedRunningTime="2026-09-10 14:28:35.44430093 +0000 UTC m=+41.870038353"
Sep 10 14:28:35.527025 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.526989    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-phvc7"]
Sep 10 14:28:35.548370 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.548343    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-phvc7"]
Sep 10 14:28:35.548492 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.548383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.550490 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.550451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 14:28:35.550490 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.550486    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 14:28:35.550663 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.550486    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 14:28:35.550834 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.550819    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fc42h\""
Sep 10 14:28:35.550913 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.550823    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 14:28:35.696833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.696745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.696833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.696796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4pb\" (UniqueName: \"kubernetes.io/projected/edc97f05-11d3-4259-803c-2e7cb859844e-kube-api-access-xh4pb\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.697016 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.696889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.697016 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.696981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edc97f05-11d3-4259-803c-2e7cb859844e-crio-socket\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.697081 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.697027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edc97f05-11d3-4259-803c-2e7cb859844e-data-volume\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.797638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.797598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edc97f05-11d3-4259-803c-2e7cb859844e-crio-socket\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.797638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.797643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edc97f05-11d3-4259-803c-2e7cb859844e-data-volume\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.797928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.797751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.797928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.797790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4pb\" (UniqueName: \"kubernetes.io/projected/edc97f05-11d3-4259-803c-2e7cb859844e-kube-api-access-xh4pb\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.797928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.797829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.797928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.797873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edc97f05-11d3-4259-803c-2e7cb859844e-crio-socket\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.797928 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:35.797925    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:28:35.798126 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:35.797932    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:35.798126 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:35.797979    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:36.297964224 +0000 UTC m=+42.723701622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:35.798126 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:35.798003    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:36.297985749 +0000 UTC m=+42.723723163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : configmap "kube-rbac-proxy" not found
Sep 10 14:28:35.798126 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.798085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edc97f05-11d3-4259-803c-2e7cb859844e-data-volume\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:35.806462 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:35.806432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4pb\" (UniqueName: \"kubernetes.io/projected/edc97f05-11d3-4259-803c-2e7cb859844e-kube-api-access-xh4pb\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:36.303906 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:36.303870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:36.304381 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:36.303978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:36.304381 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:36.304014    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:28:36.304381 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:36.304077    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:36.304381 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:36.304126    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:37.304102891 +0000 UTC m=+43.729840307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : configmap "kube-rbac-proxy" not found
Sep 10 14:28:36.304381 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:36.304147    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:37.304138144 +0000 UTC m=+43.729875547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:36.393090 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:36.393061    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:28:36.393491 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:36.393476    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/0.log"
Sep 10 14:28:36.393541 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:36.393514    2566 generic.go:356] "Generic (PLEG): container finished" podID="2d4f3935-7fa7-4cd5-aa0b-a5178e683e41" containerID="55e91b33017b92bd744696a85d6f6211ef012c9307a81b9d93f48ccc4084ad40" exitCode=255
Sep 10 14:28:36.393648 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:36.393621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bfwnd" event={"ID":"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41","Type":"ContainerDied","Data":"55e91b33017b92bd744696a85d6f6211ef012c9307a81b9d93f48ccc4084ad40"}
Sep 10 14:28:36.393796 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:36.393675    2566 scope.go:117] "RemoveContainer" containerID="7cd1e6848ef36fcddd9a87da699d089df54bab50d3eab2bdec6fd505a7703d0d"
Sep 10 14:28:36.393944 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:36.393927    2566 scope.go:117] "RemoveContainer" containerID="55e91b33017b92bd744696a85d6f6211ef012c9307a81b9d93f48ccc4084ad40"
Sep 10 14:28:36.394140 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:36.394103    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bfwnd_openshift-console-operator(2d4f3935-7fa7-4cd5-aa0b-a5178e683e41)\"" pod="openshift-console-operator/console-operator-85b454874b-bfwnd" podUID="2d4f3935-7fa7-4cd5-aa0b-a5178e683e41"
Sep 10 14:28:37.289137 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:37.289107    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5n69_352316f0-970c-43ad-befe-ffeaaad7e582/dns-node-resolver/0.log"
Sep 10 14:28:37.330579 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:37.330542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:37.330950 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.330671    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:28:37.330950 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:37.330713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:37.330950 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.330762    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:39.330741719 +0000 UTC m=+45.756479138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : configmap "kube-rbac-proxy" not found
Sep 10 14:28:37.330950 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.330848    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:37.330950 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.330918    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:39.330902585 +0000 UTC m=+45.756640005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:37.398353 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:37.398326    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:28:37.398649 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:37.398636    2566 scope.go:117] "RemoveContainer" containerID="55e91b33017b92bd744696a85d6f6211ef012c9307a81b9d93f48ccc4084ad40"
Sep 10 14:28:37.398816 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.398799    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bfwnd_openshift-console-operator(2d4f3935-7fa7-4cd5-aa0b-a5178e683e41)\"" pod="openshift-console-operator/console-operator-85b454874b-bfwnd" podUID="2d4f3935-7fa7-4cd5-aa0b-a5178e683e41"
Sep 10 14:28:37.734621 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:37.734573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:37.734858 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:37.734744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:37.734858 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.734759    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:28:37.734858 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.734834    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls podName:8113e71f-67b8-4cbb-90b1-7e85d15ad598 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:45.734813741 +0000 UTC m=+52.160551156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kj6xm" (UID: "8113e71f-67b8-4cbb-90b1-7e85d15ad598") : secret "samples-operator-tls" not found
Sep 10 14:28:37.735023 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.734887    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:37.735023 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.734944    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls podName:ee67ac4c-3579-4d26-a029-9f782d31589f nodeName:}" failed. No retries permitted until 2026-09-10 14:28:45.734931786 +0000 UTC m=+52.160669188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lnvg9" (UID: "ee67ac4c-3579-4d26-a029-9f782d31589f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:37.835451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:37.835409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:37.835623 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.835558    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:28:37.835623 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.835578    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6677bb96-ft5cx: secret "image-registry-tls" not found
Sep 10 14:28:37.835712 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:37.835634    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls podName:215e4021-d8eb-421d-8095-6e3f9c87e0bb nodeName:}" failed. No retries permitted until 2026-09-10 14:28:45.835618633 +0000 UTC m=+52.261356045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls") pod "image-registry-6677bb96-ft5cx" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb") : secret "image-registry-tls" not found
Sep 10 14:28:38.088373 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:38.088300    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fvv6t_532db11a-6291-4687-8d65-e71384f16400/node-ca/0.log"
Sep 10 14:28:38.888637 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:38.888600    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p26lr_312e6872-7076-44d1-8bbc-0c2b9b4161d5/serve-healthcheck-canary/0.log"
Sep 10 14:28:39.347967 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:39.347930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:39.348115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:39.348047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:39.348115 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:39.348053    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:28:39.348115 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:39.348113    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:43.348098444 +0000 UTC m=+49.773835842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : configmap "kube-rbac-proxy" not found
Sep 10 14:28:39.348250 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:39.348154    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:39.348250 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:39.348194    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:43.348182458 +0000 UTC m=+49.773919869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:40.222303 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:40.222261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:40.222819 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:40.222324    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:40.222819 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:40.222757    2566 scope.go:117] "RemoveContainer" containerID="55e91b33017b92bd744696a85d6f6211ef012c9307a81b9d93f48ccc4084ad40"
Sep 10 14:28:40.222970 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:40.222948    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bfwnd_openshift-console-operator(2d4f3935-7fa7-4cd5-aa0b-a5178e683e41)\"" pod="openshift-console-operator/console-operator-85b454874b-bfwnd" podUID="2d4f3935-7fa7-4cd5-aa0b-a5178e683e41"
Sep 10 14:28:43.281513 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:43.281471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:43.281930 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:43.281621    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:28:43.281930 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:43.281715    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls podName:cd04e0bc-77f9-44c0-87a9-d8123246c487 nodeName:}" failed. No retries permitted until 2026-09-10 14:28:59.281682871 +0000 UTC m=+65.707420269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls") pod "dns-default-98m8q" (UID: "cd04e0bc-77f9-44c0-87a9-d8123246c487") : secret "dns-default-metrics-tls" not found
Sep 10 14:28:43.382535 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:43.382496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:43.382725 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:43.382564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:43.382725 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:43.382651    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:43.382797 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:43.382730    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls podName:edc97f05-11d3-4259-803c-2e7cb859844e nodeName:}" failed. No retries permitted until 2026-09-10 14:28:51.38271597 +0000 UTC m=+57.808453384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phvc7" (UID: "edc97f05-11d3-4259-803c-2e7cb859844e") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:28:43.383149 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:43.383127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edc97f05-11d3-4259-803c-2e7cb859844e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:45.803505 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:45.803455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:28:45.803993 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:45.803529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:45.803993 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:45.803627    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:45.803993 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:45.803750    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls podName:ee67ac4c-3579-4d26-a029-9f782d31589f nodeName:}" failed. No retries permitted until 2026-09-10 14:29:01.803727926 +0000 UTC m=+68.229465327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lnvg9" (UID: "ee67ac4c-3579-4d26-a029-9f782d31589f") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:28:45.805972 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:45.805952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8113e71f-67b8-4cbb-90b1-7e85d15ad598-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kj6xm\" (UID: \"8113e71f-67b8-4cbb-90b1-7e85d15ad598\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:45.833140 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:45.833109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"
Sep 10 14:28:45.904914 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:45.904882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:28:45.905058 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:45.905000    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:28:45.905058 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:45.905014    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6677bb96-ft5cx: secret "image-registry-tls" not found
Sep 10 14:28:45.905139 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:28:45.905062    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls podName:215e4021-d8eb-421d-8095-6e3f9c87e0bb nodeName:}" failed. No retries permitted until 2026-09-10 14:29:01.905049769 +0000 UTC m=+68.330787180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls") pod "image-registry-6677bb96-ft5cx" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb") : secret "image-registry-tls" not found
Sep 10 14:28:46.101413 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:46.101385    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm"]
Sep 10 14:28:46.429162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:46.429126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm" event={"ID":"8113e71f-67b8-4cbb-90b1-7e85d15ad598","Type":"ContainerStarted","Data":"9a1102e3c4d6b4fa24240b6994f5ea9d7b156a0ded06cc80ffa953990c04ac22"}
Sep 10 14:28:48.437009 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:48.436970    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm" event={"ID":"8113e71f-67b8-4cbb-90b1-7e85d15ad598","Type":"ContainerStarted","Data":"57264e10926f16ca931616a7d4256f0d31ce9f2b95fc10cd86b7e87164424498"}
Sep 10 14:28:48.437501 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:48.437030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm" event={"ID":"8113e71f-67b8-4cbb-90b1-7e85d15ad598","Type":"ContainerStarted","Data":"416e20b2359bb8756496e44295ec72872065c84e9386634b7aa75aa9e1465043"}
Sep 10 14:28:48.455347 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:48.455299    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kj6xm" podStartSLOduration=17.750115501 podStartE2EDuration="19.455281002s" totalImagesPullingTime="1.705165501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:48.454327489 +0000 UTC m=+54.880064912" watchObservedRunningTime="2026-09-10 14:28:48.455281002 +0000 UTC m=+54.881018420"
Sep 10 14:28:51.453645 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:51.453612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:51.455863 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:51.455840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edc97f05-11d3-4259-803c-2e7cb859844e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phvc7\" (UID: \"edc97f05-11d3-4259-803c-2e7cb859844e\") " pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:51.456764 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:51.456744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-phvc7"
Sep 10 14:28:51.751546 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:51.751315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-phvc7"]
Sep 10 14:28:51.754101 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:28:51.754076    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc97f05_11d3_4259_803c_2e7cb859844e.slice/crio-ec96c4fec046f46057c3a2f9cbc00a7aed27bc000ba6c2a3e411d668822cef59 WatchSource:0}: Error finding container ec96c4fec046f46057c3a2f9cbc00a7aed27bc000ba6c2a3e411d668822cef59: Status 404 returned error can't find the container with id ec96c4fec046f46057c3a2f9cbc00a7aed27bc000ba6c2a3e411d668822cef59
Sep 10 14:28:52.337479 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:52.337458    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbrcr"
Sep 10 14:28:52.451310 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:52.451274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phvc7" event={"ID":"edc97f05-11d3-4259-803c-2e7cb859844e","Type":"ContainerStarted","Data":"c07120486d97183f2fd52a4d2e3e07e74d72277815a3b9462bd94f81b8a0d5f0"}
Sep 10 14:28:52.451501 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:52.451345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phvc7" event={"ID":"edc97f05-11d3-4259-803c-2e7cb859844e","Type":"ContainerStarted","Data":"088111ee1573ddabadc21962b3dbc4b49c97dcbc81d802b714b0575b0f525f39"}
Sep 10 14:28:52.451501 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:52.451372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phvc7" event={"ID":"edc97f05-11d3-4259-803c-2e7cb859844e","Type":"ContainerStarted","Data":"ec96c4fec046f46057c3a2f9cbc00a7aed27bc000ba6c2a3e411d668822cef59"}
Sep 10 14:28:53.104476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:53.104441    2566 scope.go:117] "RemoveContainer" containerID="55e91b33017b92bd744696a85d6f6211ef012c9307a81b9d93f48ccc4084ad40"
Sep 10 14:28:53.456793 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:53.456761    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:28:53.456971 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:53.456839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bfwnd" event={"ID":"2d4f3935-7fa7-4cd5-aa0b-a5178e683e41","Type":"ContainerStarted","Data":"ab50bd792ace52142afcee725bc474d889f960d994d77ba6c7554ada7e5d29e4"}
Sep 10 14:28:53.457218 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:53.457165    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:53.472061 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:53.472012    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-bfwnd" podStartSLOduration=20.274623171 podStartE2EDuration="24.471998181s" totalImagesPullingTime="4.19737501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:53.471399159 +0000 UTC m=+59.897136574" watchObservedRunningTime="2026-09-10 14:28:53.471998181 +0000 UTC m=+59.897735614"
Sep 10 14:28:53.793194 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:53.793166    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-bfwnd"
Sep 10 14:28:54.462661 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:54.462618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phvc7" event={"ID":"edc97f05-11d3-4259-803c-2e7cb859844e","Type":"ContainerStarted","Data":"f6f712ccb3c88dcca57a09d37c2db3715a912188f9e6c4f522f0724155a136ac"}
Sep 10 14:28:54.479543 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:54.479488    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-phvc7" podStartSLOduration=17.583923213 podStartE2EDuration="19.479471723s" totalImagesPullingTime="1.89554851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:28:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:28:54.478500432 +0000 UTC m=+60.904237851" watchObservedRunningTime="2026-09-10 14:28:54.479471723 +0000 UTC m=+60.905209183"
Sep 10 14:28:59.322719 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.322671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:59.325067 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.325041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd04e0bc-77f9-44c0-87a9-d8123246c487-metrics-tls\") pod \"dns-default-98m8q\" (UID: \"cd04e0bc-77f9-44c0-87a9-d8123246c487\") " pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:59.490814 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.490783    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lsf8q\""
Sep 10 14:28:59.498680 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.498659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98m8q"
Sep 10 14:28:59.727890 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.727854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:59.730146 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.730129    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:28:59.740836 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.740815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe629083-72c1-416f-a8ab-38fe211e7598-metrics-certs\") pod \"network-metrics-daemon-qfbg6\" (UID: \"fe629083-72c1-416f-a8ab-38fe211e7598\") " pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:28:59.762357 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.762327    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98m8q"]
Sep 10 14:28:59.765484 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:28:59.765460    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd04e0bc_77f9_44c0_87a9_d8123246c487.slice/crio-00413e1af2ad0a971f8fdfe293d72712608aa97eb6ed58c2aef46f601addb5ea WatchSource:0}: Error finding container 00413e1af2ad0a971f8fdfe293d72712608aa97eb6ed58c2aef46f601addb5ea: Status 404 returned error can't find the container with id 00413e1af2ad0a971f8fdfe293d72712608aa97eb6ed58c2aef46f601addb5ea
Sep 10 14:28:59.828315 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.828273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:59.830325 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.830303    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:28:59.840959 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.840940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4-original-pull-secret\") pod \"global-pull-secret-syncer-8tb9v\" (UID: \"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4\") " pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:28:59.929571 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.929534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:28:59.931954 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.931910    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:28:59.941978 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.941957    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:28:59.952922 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:28:59.952889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ds6k\" (UniqueName: \"kubernetes.io/projected/8abd9fae-29f2-42aa-a064-3c1928c21286-kube-api-access-6ds6k\") pod \"network-check-target-c7jb9\" (UID: \"8abd9fae-29f2-42aa-a064-3c1928c21286\") " pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:29:00.018124 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.018053    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pptck\""
Sep 10 14:29:00.025436 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.025411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bf25v\""
Sep 10 14:29:00.026458 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.026443    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:29:00.029717 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.029678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tb9v"
Sep 10 14:29:00.034300 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.034278    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfbg6"
Sep 10 14:29:00.300049 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.299937    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8tb9v"]
Sep 10 14:29:00.302478 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.302449    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c7jb9"]
Sep 10 14:29:00.303049 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:29:00.302913    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6390be7d_6bcc_44e6_be5b_7d4f3fdd92e4.slice/crio-69691ae219a557d6be40df29639dd483469a1347e2e92b67227707a065a7dddb WatchSource:0}: Error finding container 69691ae219a557d6be40df29639dd483469a1347e2e92b67227707a065a7dddb: Status 404 returned error can't find the container with id 69691ae219a557d6be40df29639dd483469a1347e2e92b67227707a065a7dddb
Sep 10 14:29:00.305558 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:29:00.305539    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abd9fae_29f2_42aa_a064_3c1928c21286.slice/crio-f1d40e7d20569ecab196bcc96a282b171099754b8efcbe24b72a93a0827f9fc3 WatchSource:0}: Error finding container f1d40e7d20569ecab196bcc96a282b171099754b8efcbe24b72a93a0827f9fc3: Status 404 returned error can't find the container with id f1d40e7d20569ecab196bcc96a282b171099754b8efcbe24b72a93a0827f9fc3
Sep 10 14:29:00.315221 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.315178    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qfbg6"]
Sep 10 14:29:00.319312 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:29:00.319290    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe629083_72c1_416f_a8ab_38fe211e7598.slice/crio-3956096fa7a56d6b0bff1fe287828bcf779a31aea1c90535af18d69abd15fccc WatchSource:0}: Error finding container 3956096fa7a56d6b0bff1fe287828bcf779a31aea1c90535af18d69abd15fccc: Status 404 returned error can't find the container with id 3956096fa7a56d6b0bff1fe287828bcf779a31aea1c90535af18d69abd15fccc
Sep 10 14:29:00.482834 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.482788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98m8q" event={"ID":"cd04e0bc-77f9-44c0-87a9-d8123246c487","Type":"ContainerStarted","Data":"00413e1af2ad0a971f8fdfe293d72712608aa97eb6ed58c2aef46f601addb5ea"}
Sep 10 14:29:00.484326 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.484298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8tb9v" event={"ID":"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4","Type":"ContainerStarted","Data":"69691ae219a557d6be40df29639dd483469a1347e2e92b67227707a065a7dddb"}
Sep 10 14:29:00.485874 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.485833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c7jb9" event={"ID":"8abd9fae-29f2-42aa-a064-3c1928c21286","Type":"ContainerStarted","Data":"f1d40e7d20569ecab196bcc96a282b171099754b8efcbe24b72a93a0827f9fc3"}
Sep 10 14:29:00.487190 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:00.487158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qfbg6" event={"ID":"fe629083-72c1-416f-a8ab-38fe211e7598","Type":"ContainerStarted","Data":"3956096fa7a56d6b0bff1fe287828bcf779a31aea1c90535af18d69abd15fccc"}
Sep 10 14:29:01.846750 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.846687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:29:01.851116 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.851078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee67ac4c-3579-4d26-a029-9f782d31589f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lnvg9\" (UID: \"ee67ac4c-3579-4d26-a029-9f782d31589f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:29:01.920727 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.920269    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6677bb96-ft5cx"]
Sep 10 14:29:01.920727 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:29:01.920636    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6677bb96-ft5cx" podUID="215e4021-d8eb-421d-8095-6e3f9c87e0bb"
Sep 10 14:29:01.938885 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.937646    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"]
Sep 10 14:29:01.941150 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.940978    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"]
Sep 10 14:29:01.941150 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.941125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"
Sep 10 14:29:01.945233 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.945067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:01.945361 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.945340    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 14:29:01.945739 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.945523    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xfjkt\""
Sep 10 14:29:01.945739 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.945524    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 14:29:01.946057 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.946034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 14:29:01.946255 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.946036    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 14:29:01.947293 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.947135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:29:01.947293 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.947243    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 14:29:01.948060 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.947625    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 14:29:01.948060 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.947838    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 14:29:01.950339 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.950188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"image-registry-6677bb96-ft5cx\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") " pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:29:01.954012 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.953250    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 14:29:01.955188 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.955165    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"]
Sep 10 14:29:01.961114 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:01.959592    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"]
Sep 10 14:29:02.048235 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.048003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmf4\" (UniqueName: \"kubernetes.io/projected/c11883e2-127c-420f-b9a2-940fa179a856-kube-api-access-hsmf4\") pod \"managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6\" (UID: \"c11883e2-127c-420f-b9a2-940fa179a856\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"
Sep 10 14:29:02.048235 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.048050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-hub\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.048235 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.048082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.048235 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.048124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.048235 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.048165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.048235 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.048192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-ca\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.048235 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.048217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c11883e2-127c-420f-b9a2-940fa179a856-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6\" (UID: \"c11883e2-127c-420f-b9a2-940fa179a856\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"
Sep 10 14:29:02.048685 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.048248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25csk\" (UniqueName: \"kubernetes.io/projected/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-kube-api-access-25csk\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.079357 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.079321    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pxrpp\""
Sep 10 14:29:02.087484 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.087455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"
Sep 10 14:29:02.149734 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.149611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmf4\" (UniqueName: \"kubernetes.io/projected/c11883e2-127c-420f-b9a2-940fa179a856-kube-api-access-hsmf4\") pod \"managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6\" (UID: \"c11883e2-127c-420f-b9a2-940fa179a856\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"
Sep 10 14:29:02.149734 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.149662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-hub\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.149734 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.149714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.150129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.149758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.150129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.149801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.150129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.149827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-ca\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.150129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.149848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c11883e2-127c-420f-b9a2-940fa179a856-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6\" (UID: \"c11883e2-127c-420f-b9a2-940fa179a856\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"
Sep 10 14:29:02.150129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.149876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25csk\" (UniqueName: \"kubernetes.io/projected/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-kube-api-access-25csk\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.151074 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.150755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.155409 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.154076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-hub\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.155409 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.155200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c11883e2-127c-420f-b9a2-940fa179a856-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6\" (UID: \"c11883e2-127c-420f-b9a2-940fa179a856\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"
Sep 10 14:29:02.156136 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.155635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-ca\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.162106 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.161770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.164560 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.162673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.169487 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.169298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmf4\" (UniqueName: \"kubernetes.io/projected/c11883e2-127c-420f-b9a2-940fa179a856-kube-api-access-hsmf4\") pod \"managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6\" (UID: \"c11883e2-127c-420f-b9a2-940fa179a856\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"
Sep 10 14:29:02.170560 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.170384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25csk\" (UniqueName: \"kubernetes.io/projected/3ad779ca-51be-48a9-a9d1-1ca81f43f36a-kube-api-access-25csk\") pod \"cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5\" (UID: \"3ad779ca-51be-48a9-a9d1-1ca81f43f36a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.273267 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.273195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"
Sep 10 14:29:02.280523 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.280333    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"
Sep 10 14:29:02.389112 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.389072    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9"]
Sep 10 14:29:02.392426 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:29:02.392392    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee67ac4c_3579_4d26_a029_9f782d31589f.slice/crio-0dbdc69b6d5c3d9805a9b93996a976d130b4389806e6877c088a0a6e38d729aa WatchSource:0}: Error finding container 0dbdc69b6d5c3d9805a9b93996a976d130b4389806e6877c088a0a6e38d729aa: Status 404 returned error can't find the container with id 0dbdc69b6d5c3d9805a9b93996a976d130b4389806e6877c088a0a6e38d729aa
Sep 10 14:29:02.500668 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.500629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qfbg6" event={"ID":"fe629083-72c1-416f-a8ab-38fe211e7598","Type":"ContainerStarted","Data":"82d558df5d39b35f08840e3ee97d6015749d1a08afefb7c9e7f3ecc0488bf67e"}
Sep 10 14:29:02.500877 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.500717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qfbg6" event={"ID":"fe629083-72c1-416f-a8ab-38fe211e7598","Type":"ContainerStarted","Data":"5c9ce20e392f943bf1ff5e6ca8a3d5520d05f6092d65d46d5e8825219aafc4be"}
Sep 10 14:29:02.502980 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.502950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98m8q" event={"ID":"cd04e0bc-77f9-44c0-87a9-d8123246c487","Type":"ContainerStarted","Data":"73eedbe1e1e58d12306d8fe10619b1d7efb37e83248fbc6e9c2f401526f295d0"}
Sep 10 14:29:02.503108 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.503004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98m8q" event={"ID":"cd04e0bc-77f9-44c0-87a9-d8123246c487","Type":"ContainerStarted","Data":"9b059721abf91dfc8e43720a28fe6a282023c89b6aa09f3ae8373d00a92c8c51"}
Sep 10 14:29:02.503490 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.503413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-98m8q"
Sep 10 14:29:02.504902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.504869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9" event={"ID":"ee67ac4c-3579-4d26-a029-9f782d31589f","Type":"ContainerStarted","Data":"0dbdc69b6d5c3d9805a9b93996a976d130b4389806e6877c088a0a6e38d729aa"}
Sep 10 14:29:02.504902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.504885    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:29:02.518168 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.518064    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qfbg6" podStartSLOduration=66.827492611 podStartE2EDuration="1m8.518046742s" totalImagesPullingTime="1.690554131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:02.515746373 +0000 UTC m=+68.941483793" watchObservedRunningTime="2026-09-10 14:29:02.518046742 +0000 UTC m=+68.943784162"
Sep 10 14:29:02.525296 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.525272    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:29:02.532328 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.532070    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-98m8q" podStartSLOduration=33.287076039 podStartE2EDuration="35.53205178s" totalImagesPullingTime="2.244975741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:02.531262431 +0000 UTC m=+68.956999852" watchObservedRunningTime="2026-09-10 14:29:02.53205178 +0000 UTC m=+68.957789201"
Sep 10 14:29:02.577139 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.577105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6"]
Sep 10 14:29:02.579606 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.579553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5"]
Sep 10 14:29:02.581379 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:29:02.581352    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc11883e2_127c_420f_b9a2_940fa179a856.slice/crio-5f61190fb064688046f96e37797dc13a5f95c8a3a8681120faaabc1437314436 WatchSource:0}: Error finding container 5f61190fb064688046f96e37797dc13a5f95c8a3a8681120faaabc1437314436: Status 404 returned error can't find the container with id 5f61190fb064688046f96e37797dc13a5f95c8a3a8681120faaabc1437314436
Sep 10 14:29:02.583150 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:29:02.583125    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ad779ca_51be_48a9_a9d1_1ca81f43f36a.slice/crio-a9ccfa8af445d55468c17dc3dbd046ffb770c3295b056b5287b3c73d1f6305ae WatchSource:0}: Error finding container a9ccfa8af445d55468c17dc3dbd046ffb770c3295b056b5287b3c73d1f6305ae: Status 404 returned error can't find the container with id a9ccfa8af445d55468c17dc3dbd046ffb770c3295b056b5287b3c73d1f6305ae
Sep 10 14:29:02.653471 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.653439    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-installation-pull-secrets\") pod \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") "
Sep 10 14:29:02.653666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.653485    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") pod \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") "
Sep 10 14:29:02.653666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.653524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74689\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-kube-api-access-74689\") pod \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") "
Sep 10 14:29:02.653666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.653548    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-trusted-ca\") pod \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") "
Sep 10 14:29:02.653666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.653587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-certificates\") pod \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") "
Sep 10 14:29:02.653666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.653633    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-bound-sa-token\") pod \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") "
Sep 10 14:29:02.653666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.653661    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-image-registry-private-configuration\") pod \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") "
Sep 10 14:29:02.653956 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.653721    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/215e4021-d8eb-421d-8095-6e3f9c87e0bb-ca-trust-extracted\") pod \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\" (UID: \"215e4021-d8eb-421d-8095-6e3f9c87e0bb\") "
Sep 10 14:29:02.654533 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.654497    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "215e4021-d8eb-421d-8095-6e3f9c87e0bb" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:29:02.656055 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.656022    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "215e4021-d8eb-421d-8095-6e3f9c87e0bb" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:29:02.656162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.656095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215e4021-d8eb-421d-8095-6e3f9c87e0bb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "215e4021-d8eb-421d-8095-6e3f9c87e0bb" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:29:02.657023 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.656992    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-kube-api-access-74689" (OuterVolumeSpecName: "kube-api-access-74689") pod "215e4021-d8eb-421d-8095-6e3f9c87e0bb" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb"). InnerVolumeSpecName "kube-api-access-74689". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:29:02.657023 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.657011    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "215e4021-d8eb-421d-8095-6e3f9c87e0bb" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:29:02.657748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.657722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "215e4021-d8eb-421d-8095-6e3f9c87e0bb" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:29:02.657865 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.657785    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "215e4021-d8eb-421d-8095-6e3f9c87e0bb" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:29:02.658995 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.658970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "215e4021-d8eb-421d-8095-6e3f9c87e0bb" (UID: "215e4021-d8eb-421d-8095-6e3f9c87e0bb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:29:02.754536 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.754499    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-installation-pull-secrets\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:29:02.754536 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.754534    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:29:02.754799 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.754550    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74689\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-kube-api-access-74689\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:29:02.754799 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.754564    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-trusted-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:29:02.754799 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.754578    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/215e4021-d8eb-421d-8095-6e3f9c87e0bb-registry-certificates\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:29:02.754799 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.754591    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/215e4021-d8eb-421d-8095-6e3f9c87e0bb-bound-sa-token\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:29:02.754799 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.754606    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/215e4021-d8eb-421d-8095-6e3f9c87e0bb-image-registry-private-configuration\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:29:02.754799 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:02.754619    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/215e4021-d8eb-421d-8095-6e3f9c87e0bb-ca-trust-extracted\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:29:03.510969 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:03.510926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5" event={"ID":"3ad779ca-51be-48a9-a9d1-1ca81f43f36a","Type":"ContainerStarted","Data":"a9ccfa8af445d55468c17dc3dbd046ffb770c3295b056b5287b3c73d1f6305ae"}
Sep 10 14:29:03.512597 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:03.512570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6677bb96-ft5cx"
Sep 10 14:29:03.512597 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:03.512580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6" event={"ID":"c11883e2-127c-420f-b9a2-940fa179a856","Type":"ContainerStarted","Data":"5f61190fb064688046f96e37797dc13a5f95c8a3a8681120faaabc1437314436"}
Sep 10 14:29:03.548179 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:03.548150    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6677bb96-ft5cx"]
Sep 10 14:29:03.551537 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:03.551501    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6677bb96-ft5cx"]
Sep 10 14:29:04.110475 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:04.110430    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215e4021-d8eb-421d-8095-6e3f9c87e0bb" path="/var/lib/kubelet/pods/215e4021-d8eb-421d-8095-6e3f9c87e0bb/volumes"
Sep 10 14:29:09.537885 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.537852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8tb9v" event={"ID":"6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4","Type":"ContainerStarted","Data":"5d627a4da0a9f706ab82369ba27721fbd7fa699b26db04ee6571d9f098b17d62"}
Sep 10 14:29:09.539672 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.539648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c7jb9" event={"ID":"8abd9fae-29f2-42aa-a064-3c1928c21286","Type":"ContainerStarted","Data":"5ac8d4967bd369671c1852244d66e5da6143e8e5f976039fb4362552ab5e51b7"}
Sep 10 14:29:09.539824 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.539722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:29:09.541685 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.541664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5" event={"ID":"3ad779ca-51be-48a9-a9d1-1ca81f43f36a","Type":"ContainerStarted","Data":"d773ac854bc4358d1704e37f475f038194d8139bfe126ff1d29b3b5f3d14f606"}
Sep 10 14:29:09.541824 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.541708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5" event={"ID":"3ad779ca-51be-48a9-a9d1-1ca81f43f36a","Type":"ContainerStarted","Data":"c40c6652e5b7adf8665d787158a125b8c7b2f974b7f923703b0e9a6be47dd761"}
Sep 10 14:29:09.541824 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.541731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5" event={"ID":"3ad779ca-51be-48a9-a9d1-1ca81f43f36a","Type":"ContainerStarted","Data":"e44a30694e4e2c08e57c05fb6286984d8442b87ffe4fbd85517d3b93d13c02ef"}
Sep 10 14:29:09.543190 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.543167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6" event={"ID":"c11883e2-127c-420f-b9a2-940fa179a856","Type":"ContainerStarted","Data":"9c5f7a830f9eb0a7646672defffcccb7c98e076d145fd99d459e2404e3fb3d05"}
Sep 10 14:29:09.544731 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.544686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9" event={"ID":"ee67ac4c-3579-4d26-a029-9f782d31589f","Type":"ContainerStarted","Data":"523d47b51c72bb35f397dd9fc1c5500f3f5b6c04001d282cf37ee39140db9a50"}
Sep 10 14:29:09.557468 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.557412    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8tb9v" podStartSLOduration=66.728599784 podStartE2EDuration="1m15.557394447s" totalImagesPullingTime="8.828794663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:09.556451232 +0000 UTC m=+75.982188651" watchObservedRunningTime="2026-09-10 14:29:09.557394447 +0000 UTC m=+75.983131892"
Sep 10 14:29:09.572860 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.572816    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lnvg9" podStartSLOduration=33.831666895 podStartE2EDuration="40.57280223s" totalImagesPullingTime="6.741135335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:09.571387483 +0000 UTC m=+75.997124901" watchObservedRunningTime="2026-09-10 14:29:09.57280223 +0000 UTC m=+75.998539650"
Sep 10 14:29:09.592359 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.592311    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6bf4fc88-r2tm5" podStartSLOduration=2.023252143 podStartE2EDuration="8.592295367s" totalImagesPullingTime="6.569043224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:09.591840456 +0000 UTC m=+76.017577876" watchObservedRunningTime="2026-09-10 14:29:09.592295367 +0000 UTC m=+76.018032788"
Sep 10 14:29:09.605728 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.605664    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc7cf878-sr2d6" podStartSLOduration=2.039992351 podStartE2EDuration="8.605645146s" totalImagesPullingTime="6.565652795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:09.605349456 +0000 UTC m=+76.031086888" watchObservedRunningTime="2026-09-10 14:29:09.605645146 +0000 UTC m=+76.031382567"
Sep 10 14:29:09.622723 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:09.622624    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c7jb9" podStartSLOduration=66.824461191 podStartE2EDuration="1m15.622609886s" totalImagesPullingTime="8.798148695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:09.621841794 +0000 UTC m=+76.047579217" watchObservedRunningTime="2026-09-10 14:29:09.622609886 +0000 UTC m=+76.048347502"
Sep 10 14:29:12.515342 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:12.515310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-98m8q"
Sep 10 14:29:30.123984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.123860    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nsdvn"]
Sep 10 14:29:30.130302 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.130280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.132154 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.132132    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 14:29:30.132345 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.132330    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 14:29:30.133048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.133024    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 14:29:30.133158 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.133064    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vw687\""
Sep 10 14:29:30.133158 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.133076    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 14:29:30.258418 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61536883-dc4c-4599-b76d-0b2cbca31580-metrics-client-ca\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.258418 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-sys\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.258633 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-tls\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.258633 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-root\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.258633 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-wtmp\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.258633 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-accelerators-collector-config\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.258812 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjst9\" (UniqueName: \"kubernetes.io/projected/61536883-dc4c-4599-b76d-0b2cbca31580-kube-api-access-rjst9\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.258812 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-textfile\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.258812 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.258723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.359929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.359881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-accelerators-collector-config\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.359956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjst9\" (UniqueName: \"kubernetes.io/projected/61536883-dc4c-4599-b76d-0b2cbca31580-kube-api-access-rjst9\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.359997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-textfile\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61536883-dc4c-4599-b76d-0b2cbca31580-metrics-client-ca\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-sys\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360325 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-tls\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360325 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-root\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360325 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-wtmp\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360325 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360316    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-wtmp\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360530 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-sys\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360530 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:29:30.360341    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 14:29:30.360530 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-textfile\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360530 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61536883-dc4c-4599-b76d-0b2cbca31580-root\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360530 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:29:30.360449    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-tls podName:61536883-dc4c-4599-b76d-0b2cbca31580 nodeName:}" failed. No retries permitted until 2026-09-10 14:29:30.860426942 +0000 UTC m=+97.286164353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-tls") pod "node-exporter-nsdvn" (UID: "61536883-dc4c-4599-b76d-0b2cbca31580") : secret "node-exporter-tls" not found
Sep 10 14:29:30.360804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-accelerators-collector-config\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.360855 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.360841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61536883-dc4c-4599-b76d-0b2cbca31580-metrics-client-ca\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.362638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.362612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.369890 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.369872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjst9\" (UniqueName: \"kubernetes.io/projected/61536883-dc4c-4599-b76d-0b2cbca31580-kube-api-access-rjst9\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.863279 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.863244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-tls\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:30.865663 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:30.865637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61536883-dc4c-4599-b76d-0b2cbca31580-node-exporter-tls\") pod \"node-exporter-nsdvn\" (UID: \"61536883-dc4c-4599-b76d-0b2cbca31580\") " pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:31.039998 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:31.039918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nsdvn"
Sep 10 14:29:31.055297 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:29:31.055267    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61536883_dc4c_4599_b76d_0b2cbca31580.slice/crio-f94d6e9c0c7dd35255bdabf07499207b683a66c3d96b8df323be0cc22f8379a0 WatchSource:0}: Error finding container f94d6e9c0c7dd35255bdabf07499207b683a66c3d96b8df323be0cc22f8379a0: Status 404 returned error can't find the container with id f94d6e9c0c7dd35255bdabf07499207b683a66c3d96b8df323be0cc22f8379a0
Sep 10 14:29:31.622631 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:31.622593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nsdvn" event={"ID":"61536883-dc4c-4599-b76d-0b2cbca31580","Type":"ContainerStarted","Data":"f94d6e9c0c7dd35255bdabf07499207b683a66c3d96b8df323be0cc22f8379a0"}
Sep 10 14:29:32.628439 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:32.628404    2566 generic.go:356] "Generic (PLEG): container finished" podID="61536883-dc4c-4599-b76d-0b2cbca31580" containerID="e0a45a7619f278ea99319b289bc6e876c9f824efbcdb05290d3cc98a8a10891c" exitCode=0
Sep 10 14:29:32.628942 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:32.628499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nsdvn" event={"ID":"61536883-dc4c-4599-b76d-0b2cbca31580","Type":"ContainerDied","Data":"e0a45a7619f278ea99319b289bc6e876c9f824efbcdb05290d3cc98a8a10891c"}
Sep 10 14:29:33.635002 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:33.634967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nsdvn" event={"ID":"61536883-dc4c-4599-b76d-0b2cbca31580","Type":"ContainerStarted","Data":"2514f707db2f7e3b1210e593a6242c75febcc32d26ef4e2244e5b0e39a1c8152"}
Sep 10 14:29:33.635385 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:33.635021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nsdvn" event={"ID":"61536883-dc4c-4599-b76d-0b2cbca31580","Type":"ContainerStarted","Data":"de86392d53aef9b22a03e21d31315148b6991cbd8a8c91e2e2d8bf74487889cf"}
Sep 10 14:29:33.652771 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:33.652722    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nsdvn" podStartSLOduration=2.596408435 podStartE2EDuration="3.65268041s" totalImagesPullingTime="994.074115ms" totalInitContainerRuntime="62.19786ms" isStatefulPod=true podCreationTimestamp="2026-09-10 14:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:33.651923232 +0000 UTC m=+100.077660652" watchObservedRunningTime="2026-09-10 14:29:33.65268041 +0000 UTC m=+100.078417824"
Sep 10 14:29:40.551889 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:29:40.551857    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c7jb9"
Sep 10 14:30:01.735749 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:30:01.735711    2566 generic.go:356] "Generic (PLEG): container finished" podID="7db503b0-a2c2-415b-9481-6b65ad9b7d23" containerID="295ea7d7ddc4b3d4774b3928c5d806781442f1f6b26360ec8119d3e807e1183e" exitCode=0
Sep 10 14:30:01.736155 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:30:01.735768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9" event={"ID":"7db503b0-a2c2-415b-9481-6b65ad9b7d23","Type":"ContainerDied","Data":"295ea7d7ddc4b3d4774b3928c5d806781442f1f6b26360ec8119d3e807e1183e"}
Sep 10 14:30:01.736155 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:30:01.736077    2566 scope.go:117] "RemoveContainer" containerID="295ea7d7ddc4b3d4774b3928c5d806781442f1f6b26360ec8119d3e807e1183e"
Sep 10 14:30:02.740963 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:30:02.740926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-f4rj9" event={"ID":"7db503b0-a2c2-415b-9481-6b65ad9b7d23","Type":"ContainerStarted","Data":"9185d11e00326f2a0027d921eb1946ba4d86f3a18227de9d216ad608948bed33"}
Sep 10 14:30:05.752428 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:30:05.752392    2566 generic.go:356] "Generic (PLEG): container finished" podID="96ede326-292a-4951-9185-329b2e579f71" containerID="1328ef4160fc4acba1dd49c36d1586c750f8872dc3dc79859bce24603cbd2557" exitCode=0
Sep 10 14:30:05.752832 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:30:05.752474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd" event={"ID":"96ede326-292a-4951-9185-329b2e579f71","Type":"ContainerDied","Data":"1328ef4160fc4acba1dd49c36d1586c750f8872dc3dc79859bce24603cbd2557"}
Sep 10 14:30:05.752832 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:30:05.752818    2566 scope.go:117] "RemoveContainer" containerID="1328ef4160fc4acba1dd49c36d1586c750f8872dc3dc79859bce24603cbd2557"
Sep 10 14:30:06.757871 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:30:06.757837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-tt6bd" event={"ID":"96ede326-292a-4951-9185-329b2e579f71","Type":"ContainerStarted","Data":"1910ce93bafebca87841a7235aae9bc362d75eff41516d9d13e34e27c069d5f5"}
Sep 10 14:32:34.113501 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.113470    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"]
Sep 10 14:32:34.115329 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.115313    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"
Sep 10 14:32:34.117465 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.117441    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 14:32:34.118101 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.118070    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:32:34.118171 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.118088    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-khg79\""
Sep 10 14:32:34.124446 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.124425    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"]
Sep 10 14:32:34.174157 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.174118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzf7\" (UniqueName: \"kubernetes.io/projected/b5e80416-fa13-4949-864e-0d14149e7d70-kube-api-access-9wzf7\") pod \"openshift-lws-operator-fd8ccff4c-swrr5\" (UID: \"b5e80416-fa13-4949-864e-0d14149e7d70\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"
Sep 10 14:32:34.174310 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.174177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e80416-fa13-4949-864e-0d14149e7d70-tmp\") pod \"openshift-lws-operator-fd8ccff4c-swrr5\" (UID: \"b5e80416-fa13-4949-864e-0d14149e7d70\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"
Sep 10 14:32:34.274937 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.274898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzf7\" (UniqueName: \"kubernetes.io/projected/b5e80416-fa13-4949-864e-0d14149e7d70-kube-api-access-9wzf7\") pod \"openshift-lws-operator-fd8ccff4c-swrr5\" (UID: \"b5e80416-fa13-4949-864e-0d14149e7d70\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"
Sep 10 14:32:34.275108 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.274955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e80416-fa13-4949-864e-0d14149e7d70-tmp\") pod \"openshift-lws-operator-fd8ccff4c-swrr5\" (UID: \"b5e80416-fa13-4949-864e-0d14149e7d70\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"
Sep 10 14:32:34.275429 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.275408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e80416-fa13-4949-864e-0d14149e7d70-tmp\") pod \"openshift-lws-operator-fd8ccff4c-swrr5\" (UID: \"b5e80416-fa13-4949-864e-0d14149e7d70\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"
Sep 10 14:32:34.282862 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.282833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzf7\" (UniqueName: \"kubernetes.io/projected/b5e80416-fa13-4949-864e-0d14149e7d70-kube-api-access-9wzf7\") pod \"openshift-lws-operator-fd8ccff4c-swrr5\" (UID: \"b5e80416-fa13-4949-864e-0d14149e7d70\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"
Sep 10 14:32:34.424340 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.424297    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"
Sep 10 14:32:34.692365 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:34.692285    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5"]
Sep 10 14:32:34.695205 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:32:34.695178    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5e80416_fa13_4949_864e_0d14149e7d70.slice/crio-1946b1df1ab393a01002cb9792191b40fed597e2fb848d3076a46dd4763708a7 WatchSource:0}: Error finding container 1946b1df1ab393a01002cb9792191b40fed597e2fb848d3076a46dd4763708a7: Status 404 returned error can't find the container with id 1946b1df1ab393a01002cb9792191b40fed597e2fb848d3076a46dd4763708a7
Sep 10 14:32:35.273825 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:35.273791    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5" event={"ID":"b5e80416-fa13-4949-864e-0d14149e7d70","Type":"ContainerStarted","Data":"1946b1df1ab393a01002cb9792191b40fed597e2fb848d3076a46dd4763708a7"}
Sep 10 14:32:38.287398 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:38.287362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5" event={"ID":"b5e80416-fa13-4949-864e-0d14149e7d70","Type":"ContainerStarted","Data":"0e9cebdbcf7bae9249a4dacfcc0d25b658d3a62dc50623bd38df584844afc0ff"}
Sep 10 14:32:38.305259 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:38.305203    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-swrr5" podStartSLOduration=1.720021317 podStartE2EDuration="4.305186808s" totalImagesPullingTime="2.585165491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:38.304084172 +0000 UTC m=+284.729821584" watchObservedRunningTime="2026-09-10 14:32:38.305186808 +0000 UTC m=+284.730924220"
Sep 10 14:32:53.966366 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:53.966333    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:32:53.969606 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:53.969571    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:32:53.970368 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:53.970349    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:32:53.973494 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:53.973478    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:32:53.981981 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:53.978774    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 14:32:55.900367 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:55.900329    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"]
Sep 10 14:32:55.903535 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:55.903517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:55.906226 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:55.906206    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2nvw6\""
Sep 10 14:32:55.906523 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:55.906504    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 14:32:55.906725 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:55.906711    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 14:32:55.906968 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:55.906950    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 14:32:55.907016 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:55.906989    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 14:32:55.920880 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:55.920849    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"]
Sep 10 14:32:56.037431 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.037389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b513491b-fb81-4782-ae5a-1cf7305e88c3-tmp\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.037614 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.037440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b513491b-fb81-4782-ae5a-1cf7305e88c3-manifests\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.037614 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.037468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdlzl\" (UniqueName: \"kubernetes.io/projected/b513491b-fb81-4782-ae5a-1cf7305e88c3-kube-api-access-sdlzl\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.037614 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.037499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b513491b-fb81-4782-ae5a-1cf7305e88c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.037614 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.037516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b513491b-fb81-4782-ae5a-1cf7305e88c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.138948 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.138911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b513491b-fb81-4782-ae5a-1cf7305e88c3-manifests\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.138948 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.138952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdlzl\" (UniqueName: \"kubernetes.io/projected/b513491b-fb81-4782-ae5a-1cf7305e88c3-kube-api-access-sdlzl\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.139209 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.138977    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b513491b-fb81-4782-ae5a-1cf7305e88c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.139209 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.139016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b513491b-fb81-4782-ae5a-1cf7305e88c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.139209 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.139090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b513491b-fb81-4782-ae5a-1cf7305e88c3-tmp\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.139495 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.139472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b513491b-fb81-4782-ae5a-1cf7305e88c3-manifests\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.141475 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.141447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b513491b-fb81-4782-ae5a-1cf7305e88c3-tmp\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.141674 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.141656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b513491b-fb81-4782-ae5a-1cf7305e88c3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.141736 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.141675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b513491b-fb81-4782-ae5a-1cf7305e88c3-webhook-cert\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.148128 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.148100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdlzl\" (UniqueName: \"kubernetes.io/projected/b513491b-fb81-4782-ae5a-1cf7305e88c3-kube-api-access-sdlzl\") pod \"opendatahub-operator-controller-manager-77dd699645-w7fqh\" (UID: \"b513491b-fb81-4782-ae5a-1cf7305e88c3\") " pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.214165 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.214068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:32:56.488019 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.487992    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"]
Sep 10 14:32:56.490632 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:32:56.490604    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb513491b_fb81_4782_ae5a_1cf7305e88c3.slice/crio-690302aa993c953db50a62113f5acd7dae89d11d0af45b802ba1fccd8c0ee0ed WatchSource:0}: Error finding container 690302aa993c953db50a62113f5acd7dae89d11d0af45b802ba1fccd8c0ee0ed: Status 404 returned error can't find the container with id 690302aa993c953db50a62113f5acd7dae89d11d0af45b802ba1fccd8c0ee0ed
Sep 10 14:32:56.492801 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:56.492783    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:32:57.364039 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:32:57.364001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh" event={"ID":"b513491b-fb81-4782-ae5a-1cf7305e88c3","Type":"ContainerStarted","Data":"690302aa993c953db50a62113f5acd7dae89d11d0af45b802ba1fccd8c0ee0ed"}
Sep 10 14:33:00.380031 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:00.379990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh" event={"ID":"b513491b-fb81-4782-ae5a-1cf7305e88c3","Type":"ContainerStarted","Data":"651900e9d99ba4d0fb50361c45cd94a421b31ea8b5128aec56a5839d23b62f01"}
Sep 10 14:33:01.385626 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:01.385594    2566 generic.go:356] "Generic (PLEG): container finished" podID="b513491b-fb81-4782-ae5a-1cf7305e88c3" containerID="651900e9d99ba4d0fb50361c45cd94a421b31ea8b5128aec56a5839d23b62f01" exitCode=0
Sep 10 14:33:01.386089 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:01.385676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh" event={"ID":"b513491b-fb81-4782-ae5a-1cf7305e88c3","Type":"ContainerDied","Data":"651900e9d99ba4d0fb50361c45cd94a421b31ea8b5128aec56a5839d23b62f01"}
Sep 10 14:33:02.391804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:02.391766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh" event={"ID":"b513491b-fb81-4782-ae5a-1cf7305e88c3","Type":"ContainerStarted","Data":"fef72747c608df9b2a12360fde88aca9bd34eae176df333343eca48369176a2b"}
Sep 10 14:33:02.392208 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:02.391869    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:33:02.411622 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:02.411567    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh" podStartSLOduration=2.964676863 podStartE2EDuration="7.411546514s" totalImagesPullingTime="3.376107544s" totalInitContainerRuntime="1.070762107s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:02.410240008 +0000 UTC m=+308.835977428" watchObservedRunningTime="2026-09-10 14:33:02.411546514 +0000 UTC m=+308.837283934"
Sep 10 14:33:13.397592 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:13.397560    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77dd699645-w7fqh"
Sep 10 14:33:15.752158 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.752123    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"]
Sep 10 14:33:15.763264 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.763237    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.766231 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.766199    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 14:33:15.766748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.766659    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 14:33:15.766748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.766739    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pxcmb\""
Sep 10 14:33:15.766915 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.766661    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 14:33:15.767091 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.767068    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 14:33:15.767377 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.767327    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"]
Sep 10 14:33:15.791901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.791866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5jn\" (UniqueName: \"kubernetes.io/projected/aa103c51-36e2-4470-b069-322f7b5fdc25-kube-api-access-2h5jn\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.792062 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.791919    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa103c51-36e2-4470-b069-322f7b5fdc25-tls-certs\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.792062 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.791967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa103c51-36e2-4470-b069-322f7b5fdc25-tmp\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.893024 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.892980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5jn\" (UniqueName: \"kubernetes.io/projected/aa103c51-36e2-4470-b069-322f7b5fdc25-kube-api-access-2h5jn\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.893248 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.893076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa103c51-36e2-4470-b069-322f7b5fdc25-tls-certs\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.893248 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.893109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa103c51-36e2-4470-b069-322f7b5fdc25-tmp\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.895911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.895860    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa103c51-36e2-4470-b069-322f7b5fdc25-tmp\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.896037 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.895993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa103c51-36e2-4470-b069-322f7b5fdc25-tls-certs\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:15.901788 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:15.901760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5jn\" (UniqueName: \"kubernetes.io/projected/aa103c51-36e2-4470-b069-322f7b5fdc25-kube-api-access-2h5jn\") pod \"kube-auth-proxy-6c68946dcf-7wmsv\" (UID: \"aa103c51-36e2-4470-b069-322f7b5fdc25\") " pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:16.076005 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:16.075915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"
Sep 10 14:33:16.347240 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:16.347150    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv"]
Sep 10 14:33:16.354645 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:33:16.354614    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa103c51_36e2_4470_b069_322f7b5fdc25.slice/crio-938ce0b34aa474a715637df627fe063f93af2761d28f1e2ae0710a1e59fa6749 WatchSource:0}: Error finding container 938ce0b34aa474a715637df627fe063f93af2761d28f1e2ae0710a1e59fa6749: Status 404 returned error can't find the container with id 938ce0b34aa474a715637df627fe063f93af2761d28f1e2ae0710a1e59fa6749
Sep 10 14:33:16.454309 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:16.454271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv" event={"ID":"aa103c51-36e2-4470-b069-322f7b5fdc25","Type":"ContainerStarted","Data":"938ce0b34aa474a715637df627fe063f93af2761d28f1e2ae0710a1e59fa6749"}
Sep 10 14:33:19.471521 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:19.471484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv" event={"ID":"aa103c51-36e2-4470-b069-322f7b5fdc25","Type":"ContainerStarted","Data":"9ae993f6203d5812760b8f541c5ca445f39a6d5b5539446666b43e2ac74bf187"}
Sep 10 14:33:19.488924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:19.488840    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c68946dcf-7wmsv" podStartSLOduration=1.853730284 podStartE2EDuration="4.488822147s" totalImagesPullingTime="2.635091863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:19.487661003 +0000 UTC m=+325.913398422" watchObservedRunningTime="2026-09-10 14:33:19.488822147 +0000 UTC m=+325.914559566"
Sep 10 14:33:32.816772 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.816738    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"]
Sep 10 14:33:32.821583 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.821561    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:32.823770 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.823742    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 14:33:32.823770 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.823755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 14:33:32.823951 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.823755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 14:33:32.823951 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.823798    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dh959\""
Sep 10 14:33:32.823951 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.823910    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 14:33:32.832304 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.832279    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"]
Sep 10 14:33:32.941533 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.941495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df48f725-b7ab-46c1-8941-0eb09a65fbdc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:32.941533 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.941543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:32.941907 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.941560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:32.941907 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.941584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzpp\" (UniqueName: \"kubernetes.io/projected/df48f725-b7ab-46c1-8941-0eb09a65fbdc-kube-api-access-tfzpp\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:32.941907 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.941673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:32.941907 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.941791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:32.941907 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:32.941818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.042549 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.042504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.042549 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.042550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzpp\" (UniqueName: \"kubernetes.io/projected/df48f725-b7ab-46c1-8941-0eb09a65fbdc-kube-api-access-tfzpp\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.042804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.042578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.042804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.042606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.042804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.042621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.042804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.042655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df48f725-b7ab-46c1-8941-0eb09a65fbdc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.042804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.042680    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.043472 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.043440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.045037 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.045013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.045138 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.045043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df48f725-b7ab-46c1-8941-0eb09a65fbdc-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.045138 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.045085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.045138 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.045104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df48f725-b7ab-46c1-8941-0eb09a65fbdc-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.063122 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.063097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df48f725-b7ab-46c1-8941-0eb09a65fbdc-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.063224 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.063155    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzpp\" (UniqueName: \"kubernetes.io/projected/df48f725-b7ab-46c1-8941-0eb09a65fbdc-kube-api-access-tfzpp\") pod \"istiod-openshift-gateway-5677ccb49b-29hxg\" (UID: \"df48f725-b7ab-46c1-8941-0eb09a65fbdc\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.132022 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.131922    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:33.405222 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.405192    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"]
Sep 10 14:33:33.407553 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:33:33.407524    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf48f725_b7ab_46c1_8941_0eb09a65fbdc.slice/crio-d18b1982ea0006c19b14974475604a3fe409bd61708edc969a80218a7cde6124 WatchSource:0}: Error finding container d18b1982ea0006c19b14974475604a3fe409bd61708edc969a80218a7cde6124: Status 404 returned error can't find the container with id d18b1982ea0006c19b14974475604a3fe409bd61708edc969a80218a7cde6124
Sep 10 14:33:33.529590 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:33.529552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg" event={"ID":"df48f725-b7ab-46c1-8941-0eb09a65fbdc","Type":"ContainerStarted","Data":"d18b1982ea0006c19b14974475604a3fe409bd61708edc969a80218a7cde6124"}
Sep 10 14:33:35.439019 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.438983    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"]
Sep 10 14:33:35.445777 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.445747    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.451370 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.451295    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 14:33:35.451370 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.451326    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wn9lt\""
Sep 10 14:33:35.451625 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.451326    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 14:33:35.452714 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.452657    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 14:33:35.455274 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.455161    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"]
Sep 10 14:33:35.566984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.566948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb78ed6-059d-47a7-a246-f78fbf5299a6-metrics-cert\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.566984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.566995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgvbh\" (UniqueName: \"kubernetes.io/projected/eeb78ed6-059d-47a7-a246-f78fbf5299a6-kube-api-access-qgvbh\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.567223 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.567085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eeb78ed6-059d-47a7-a246-f78fbf5299a6-manager-config\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.567223 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.567116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeb78ed6-059d-47a7-a246-f78fbf5299a6-cert\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.667926 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.667890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb78ed6-059d-47a7-a246-f78fbf5299a6-metrics-cert\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.668115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.667941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgvbh\" (UniqueName: \"kubernetes.io/projected/eeb78ed6-059d-47a7-a246-f78fbf5299a6-kube-api-access-qgvbh\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.668115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.668015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eeb78ed6-059d-47a7-a246-f78fbf5299a6-manager-config\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.668115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.668043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeb78ed6-059d-47a7-a246-f78fbf5299a6-cert\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.669314 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.669260    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eeb78ed6-059d-47a7-a246-f78fbf5299a6-manager-config\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.671087 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.671048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeb78ed6-059d-47a7-a246-f78fbf5299a6-cert\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.671245 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.671109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb78ed6-059d-47a7-a246-f78fbf5299a6-metrics-cert\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.682341 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.682311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgvbh\" (UniqueName: \"kubernetes.io/projected/eeb78ed6-059d-47a7-a246-f78fbf5299a6-kube-api-access-qgvbh\") pod \"lws-controller-manager-7c4b8b8b58-88xlh\" (UID: \"eeb78ed6-059d-47a7-a246-f78fbf5299a6\") " pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.758516 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.758437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:35.780607 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.780568    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:33:35.780800 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:35.780650    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:33:36.061731 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:36.061702    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"]
Sep 10 14:33:36.064118 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:33:36.064090    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb78ed6_059d_47a7_a246_f78fbf5299a6.slice/crio-4f7629bcd7ccbfa226b55cef3aa66b5c5b5d0c8d6351368a3fa5b3855417c017 WatchSource:0}: Error finding container 4f7629bcd7ccbfa226b55cef3aa66b5c5b5d0c8d6351368a3fa5b3855417c017: Status 404 returned error can't find the container with id 4f7629bcd7ccbfa226b55cef3aa66b5c5b5d0c8d6351368a3fa5b3855417c017
Sep 10 14:33:36.544882 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:36.544838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh" event={"ID":"eeb78ed6-059d-47a7-a246-f78fbf5299a6","Type":"ContainerStarted","Data":"4f7629bcd7ccbfa226b55cef3aa66b5c5b5d0c8d6351368a3fa5b3855417c017"}
Sep 10 14:33:36.546297 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:36.546273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg" event={"ID":"df48f725-b7ab-46c1-8941-0eb09a65fbdc","Type":"ContainerStarted","Data":"4e618a376d54fe3ace6c45a0c3376f0bcec6beddd531207b0c5ae969c5ea4836"}
Sep 10 14:33:36.546443 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:36.546425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:36.548010 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:36.547992    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg"
Sep 10 14:33:36.567845 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:36.567793    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-29hxg" podStartSLOduration=2.196958601 podStartE2EDuration="4.567777386s" totalImagesPullingTime="2.370818785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:36.565943451 +0000 UTC m=+342.991680871" watchObservedRunningTime="2026-09-10 14:33:36.567777386 +0000 UTC m=+342.993514805"
Sep 10 14:33:38.555957 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:38.555917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh" event={"ID":"eeb78ed6-059d-47a7-a246-f78fbf5299a6","Type":"ContainerStarted","Data":"d34b21d2ad1b9c90b4afaba79af434d69fa908d648df49b90cf126a6c44dbe6f"}
Sep 10 14:33:38.573553 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:38.573500    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh" podStartSLOduration=1.26659292 podStartE2EDuration="3.573483989s" totalImagesPullingTime="2.306891069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:38.572049558 +0000 UTC m=+344.997786979" watchObservedRunningTime="2026-09-10 14:33:38.573483989 +0000 UTC m=+344.999221409"
Sep 10 14:33:39.560540 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:39.560510    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:33:50.566906 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:33:50.566832    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c4b8b8b58-88xlh"
Sep 10 14:37:53.993491 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:37:53.993464    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:37:53.996394 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:37:53.996373    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:37:53.997520 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:37:53.997502    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:37:54.000348 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:37:54.000329    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:40:30.782267 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.782234    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"]
Sep 10 14:40:30.785400 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.785377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:30.787606 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.787584    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-5b0323\""
Sep 10 14:40:30.787794 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.787653    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-5b0323-dockercfg-st6z9\""
Sep 10 14:40:30.794871 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.794847    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"]
Sep 10 14:40:30.895533 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.895500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ade8636-3258-4ddb-90bc-1e5f91ce4093-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x\" (UID: \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:30.895706 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.895580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2zb\" (UniqueName: \"kubernetes.io/projected/9ade8636-3258-4ddb-90bc-1e5f91ce4093-kube-api-access-hs2zb\") pod \"payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x\" (UID: \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:30.996870 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.996778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2zb\" (UniqueName: \"kubernetes.io/projected/9ade8636-3258-4ddb-90bc-1e5f91ce4093-kube-api-access-hs2zb\") pod \"payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x\" (UID: \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:30.996870 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.996848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ade8636-3258-4ddb-90bc-1e5f91ce4093-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x\" (UID: \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:30.997555 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:30.997529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ade8636-3258-4ddb-90bc-1e5f91ce4093-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x\" (UID: \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:31.004496 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.004471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2zb\" (UniqueName: \"kubernetes.io/projected/9ade8636-3258-4ddb-90bc-1e5f91ce4093-kube-api-access-hs2zb\") pod \"payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x\" (UID: \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:31.096929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.096894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:31.366542 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.366500    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"]
Sep 10 14:40:31.369155 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:40:31.369126    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ade8636_3258_4ddb_90bc_1e5f91ce4093.slice/crio-72aea228be3ad1e0f62887c213f4accff103c1b2f2d3c6643e36fab3e6f81a8a WatchSource:0}: Error finding container 72aea228be3ad1e0f62887c213f4accff103c1b2f2d3c6643e36fab3e6f81a8a: Status 404 returned error can't find the container with id 72aea228be3ad1e0f62887c213f4accff103c1b2f2d3c6643e36fab3e6f81a8a
Sep 10 14:40:31.370990 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.370970    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:40:31.902585 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.902552    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"]
Sep 10 14:40:31.906752 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.906735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:31.908858 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.908837    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9b1e67\""
Sep 10 14:40:31.908973 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.908864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9b1e67-dockercfg-xn2p5\""
Sep 10 14:40:31.916760 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:31.916738    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"]
Sep 10 14:40:32.006025 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.005988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jq9\" (UniqueName: \"kubernetes.io/projected/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-kube-api-access-68jq9\") pod \"payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp\" (UID: \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:32.006191 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.006120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp\" (UID: \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:32.107250 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.107208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jq9\" (UniqueName: \"kubernetes.io/projected/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-kube-api-access-68jq9\") pod \"payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp\" (UID: \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:32.107440 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.107300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp\" (UID: \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:32.108177 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.108152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp\" (UID: \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:32.118151 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.118130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jq9\" (UniqueName: \"kubernetes.io/projected/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-kube-api-access-68jq9\") pod \"payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp\" (UID: \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:32.217989 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.217900    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:32.255325 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.255257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x" event={"ID":"9ade8636-3258-4ddb-90bc-1e5f91ce4093","Type":"ContainerStarted","Data":"72aea228be3ad1e0f62887c213f4accff103c1b2f2d3c6643e36fab3e6f81a8a"}
Sep 10 14:40:32.505606 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.505525    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"]
Sep 10 14:40:32.509408 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:40:32.509373    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011df5f1_8acb_4cbd_979c_def8f6e0d1ff.slice/crio-581ff89350f671da8cb11a0077553d2aecce756af3f7a3dc3c33b3315aaa057e WatchSource:0}: Error finding container 581ff89350f671da8cb11a0077553d2aecce756af3f7a3dc3c33b3315aaa057e: Status 404 returned error can't find the container with id 581ff89350f671da8cb11a0077553d2aecce756af3f7a3dc3c33b3315aaa057e
Sep 10 14:40:32.862059 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.861983    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"]
Sep 10 14:40:32.868499 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.868472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:32.870873 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.870836    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5121f4-dockercfg-8v454\""
Sep 10 14:40:32.871063 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.871038    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5121f4\""
Sep 10 14:40:32.876798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.876764    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"]
Sep 10 14:40:32.913095 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.913062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6\" (UID: \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:32.913503 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:32.913113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tfx\" (UniqueName: \"kubernetes.io/projected/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-kube-api-access-q9tfx\") pod \"payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6\" (UID: \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:33.013651 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:33.013600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6\" (UID: \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:33.013857 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:33.013685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tfx\" (UniqueName: \"kubernetes.io/projected/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-kube-api-access-q9tfx\") pod \"payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6\" (UID: \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:33.014740 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:33.014712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6\" (UID: \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:33.021449 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:33.021415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tfx\" (UniqueName: \"kubernetes.io/projected/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-kube-api-access-q9tfx\") pod \"payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6\" (UID: \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:33.182644 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:33.182601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:33.261625 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:33.261591    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp" event={"ID":"011df5f1-8acb-4cbd-979c-def8f6e0d1ff","Type":"ContainerStarted","Data":"581ff89350f671da8cb11a0077553d2aecce756af3f7a3dc3c33b3315aaa057e"}
Sep 10 14:40:34.005021 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.004943    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"]
Sep 10 14:40:34.008325 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:40:34.008297    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaa519d_3dac_4f6d_86ba_0ea3e0d84636.slice/crio-6ddf7bc0349946ef19fed20a71a7fdaf2889caa9a25f8c897d08b4a47c224875 WatchSource:0}: Error finding container 6ddf7bc0349946ef19fed20a71a7fdaf2889caa9a25f8c897d08b4a47c224875: Status 404 returned error can't find the container with id 6ddf7bc0349946ef19fed20a71a7fdaf2889caa9a25f8c897d08b4a47c224875
Sep 10 14:40:34.268244 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.268154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp" event={"ID":"011df5f1-8acb-4cbd-979c-def8f6e0d1ff","Type":"ContainerStarted","Data":"8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336"}
Sep 10 14:40:34.269024 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.268999    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:34.271262 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.271190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6" event={"ID":"baaa519d-3dac-4f6d-86ba-0ea3e0d84636","Type":"ContainerStarted","Data":"f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66"}
Sep 10 14:40:34.271378 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.271272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6" event={"ID":"baaa519d-3dac-4f6d-86ba-0ea3e0d84636","Type":"ContainerStarted","Data":"6ddf7bc0349946ef19fed20a71a7fdaf2889caa9a25f8c897d08b4a47c224875"}
Sep 10 14:40:34.271378 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.271306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:34.273376 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.273304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x" event={"ID":"9ade8636-3258-4ddb-90bc-1e5f91ce4093","Type":"ContainerStarted","Data":"037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2"}
Sep 10 14:40:34.273489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.273433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:34.294640 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.294333    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp" podStartSLOduration=2.00263019 podStartE2EDuration="3.294314244s" totalImagesPullingTime="1.291684054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:34.293202635 +0000 UTC m=+760.718940058" watchObservedRunningTime="2026-09-10 14:40:34.294314244 +0000 UTC m=+760.720051664"
Sep 10 14:40:34.315502 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.314871    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x" podStartSLOduration=1.882993906 podStartE2EDuration="4.314850735s" totalImagesPullingTime="2.431856829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:34.312556632 +0000 UTC m=+760.738294058" watchObservedRunningTime="2026-09-10 14:40:34.314850735 +0000 UTC m=+760.740588157"
Sep 10 14:40:34.335169 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:34.335111    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6" podStartSLOduration=2.335092645 podStartE2EDuration="2.335092645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:34.33186095 +0000 UTC m=+760.757598370" watchObservedRunningTime="2026-09-10 14:40:34.335092645 +0000 UTC m=+760.760830066"
Sep 10 14:40:36.972472 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:36.972432    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7830-90e4b2"]
Sep 10 14:40:36.975933 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:36.975916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7830-90e4b2"
Sep 10 14:40:36.977845 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:36.977816    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x6mfq\""
Sep 10 14:40:36.981460 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:36.981436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7830-90e4b2"]
Sep 10 14:40:37.046909 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:37.046875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4nd\" (UniqueName: \"kubernetes.io/projected/54bdc124-9ba8-4a4e-bf54-307d889b62ae-kube-api-access-4n4nd\") pod \"test-curl-7830-90e4b2\" (UID: \"54bdc124-9ba8-4a4e-bf54-307d889b62ae\") " pod="opendatahub/test-curl-7830-90e4b2"
Sep 10 14:40:37.148243 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:37.148206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4nd\" (UniqueName: \"kubernetes.io/projected/54bdc124-9ba8-4a4e-bf54-307d889b62ae-kube-api-access-4n4nd\") pod \"test-curl-7830-90e4b2\" (UID: \"54bdc124-9ba8-4a4e-bf54-307d889b62ae\") " pod="opendatahub/test-curl-7830-90e4b2"
Sep 10 14:40:37.155475 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:37.155447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4nd\" (UniqueName: \"kubernetes.io/projected/54bdc124-9ba8-4a4e-bf54-307d889b62ae-kube-api-access-4n4nd\") pod \"test-curl-7830-90e4b2\" (UID: \"54bdc124-9ba8-4a4e-bf54-307d889b62ae\") " pod="opendatahub/test-curl-7830-90e4b2"
Sep 10 14:40:37.286129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:37.286049    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7830-90e4b2"
Sep 10 14:40:37.558648 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:37.558545    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7830-90e4b2"]
Sep 10 14:40:37.561429 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:40:37.561400    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54bdc124_9ba8_4a4e_bf54_307d889b62ae.slice/crio-5ef4f0b1533bf33eb00b7d2b3e227b51662ecfd660efd05c42eac63b4a31295a WatchSource:0}: Error finding container 5ef4f0b1533bf33eb00b7d2b3e227b51662ecfd660efd05c42eac63b4a31295a: Status 404 returned error can't find the container with id 5ef4f0b1533bf33eb00b7d2b3e227b51662ecfd660efd05c42eac63b4a31295a
Sep 10 14:40:38.291751 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:38.291715    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7830-90e4b2" event={"ID":"54bdc124-9ba8-4a4e-bf54-307d889b62ae","Type":"ContainerStarted","Data":"5ef4f0b1533bf33eb00b7d2b3e227b51662ecfd660efd05c42eac63b4a31295a"}
Sep 10 14:40:40.303113 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:40.303081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7830-90e4b2" event={"ID":"54bdc124-9ba8-4a4e-bf54-307d889b62ae","Type":"ContainerStarted","Data":"145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1"}
Sep 10 14:40:40.317405 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:40.317362    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7830-90e4b2" podStartSLOduration=2.1508325680000002 podStartE2EDuration="4.317343776s" totalImagesPullingTime="2.166511208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:40.31588976 +0000 UTC m=+766.741627177" watchObservedRunningTime="2026-09-10 14:40:40.317343776 +0000 UTC m=+766.743081196"
Sep 10 14:40:40.932407 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:40.932374    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7830-90e4b2"]
Sep 10 14:40:40.940211 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:40.940183    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7830-90e4b2"]
Sep 10 14:40:41.307570 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:40:41.307533    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7830-90e4b2" container="test-curl-7830-90e4b2"
Sep 10 14:40:42.311315 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:42.311274    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7830-90e4b2" podUID="54bdc124-9ba8-4a4e-bf54-307d889b62ae" containerName="test-curl-7830-90e4b2" containerID="cri-o://145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1" gracePeriod=2
Sep 10 14:40:44.477459 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:44.477427    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7830-90e4b2"
Sep 10 14:40:44.611595 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:44.611510    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n4nd\" (UniqueName: \"kubernetes.io/projected/54bdc124-9ba8-4a4e-bf54-307d889b62ae-kube-api-access-4n4nd\") pod \"54bdc124-9ba8-4a4e-bf54-307d889b62ae\" (UID: \"54bdc124-9ba8-4a4e-bf54-307d889b62ae\") "
Sep 10 14:40:44.613667 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:44.613643    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bdc124-9ba8-4a4e-bf54-307d889b62ae-kube-api-access-4n4nd" (OuterVolumeSpecName: "kube-api-access-4n4nd") pod "54bdc124-9ba8-4a4e-bf54-307d889b62ae" (UID: "54bdc124-9ba8-4a4e-bf54-307d889b62ae"). InnerVolumeSpecName "kube-api-access-4n4nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:40:44.712105 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:44.712073    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n4nd\" (UniqueName: \"kubernetes.io/projected/54bdc124-9ba8-4a4e-bf54-307d889b62ae-kube-api-access-4n4nd\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:40:45.280013 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:45.279978    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:40:45.280172 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:45.280143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:40:45.326927 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:45.326892    2566 generic.go:356] "Generic (PLEG): container finished" podID="54bdc124-9ba8-4a4e-bf54-307d889b62ae" containerID="145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1" exitCode=137
Sep 10 14:40:45.327100 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:45.326984    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7830-90e4b2"
Sep 10 14:40:45.327100 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:45.327001    2566 scope.go:117] "RemoveContainer" containerID="145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1"
Sep 10 14:40:45.337399 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:45.337375    2566 scope.go:117] "RemoveContainer" containerID="145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1"
Sep 10 14:40:45.337760 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:40:45.337733    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1\": container with ID starting with 145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1 not found: ID does not exist" containerID="145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1"
Sep 10 14:40:45.337876 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:45.337771    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1"} err="failed to get container status \"145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1\": rpc error: code = NotFound desc = could not find container \"145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1\": container with ID starting with 145b99b8b29e78f1652368bf7b30704d5eef35615e9085ce3fc5a628311a27f1 not found: ID does not exist"
Sep 10 14:40:45.338004 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:40:45.337982    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7830-90e4b2\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="54bdc124-9ba8-4a4e-bf54-307d889b62ae" pod="opendatahub/test-curl-7830-90e4b2"
Sep 10 14:40:46.110434 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.110399    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54bdc124-9ba8-4a4e-bf54-307d889b62ae" path="/var/lib/kubelet/pods/54bdc124-9ba8-4a4e-bf54-307d889b62ae/volumes"
Sep 10 14:40:46.283650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.283624    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:40:46.534239 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.534201    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"]
Sep 10 14:40:46.534667 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.534648    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bdc124-9ba8-4a4e-bf54-307d889b62ae" containerName="test-curl-7830-90e4b2"
Sep 10 14:40:46.534721 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.534673    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bdc124-9ba8-4a4e-bf54-307d889b62ae" containerName="test-curl-7830-90e4b2"
Sep 10 14:40:46.534795 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.534783    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="54bdc124-9ba8-4a4e-bf54-307d889b62ae" containerName="test-curl-7830-90e4b2"
Sep 10 14:40:46.539377 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.539353    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.541757 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.541730    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:40:46.542009 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.541991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-22fb28-serving-cert\""
Sep 10 14:40:46.542082 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.542062    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 14:40:46.542082 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.542068    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7j4nl\""
Sep 10 14:40:46.542342 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.542327    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 14:40:46.549716 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.549670    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"]
Sep 10 14:40:46.627968 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.627933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.628104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.628012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvrh\" (UniqueName: \"kubernetes.io/projected/014799ac-9aae-4f69-aea7-31d88d380488-kube-api-access-dnvrh\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.628104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.628061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-metrics-certs\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.628104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.628101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014799ac-9aae-4f69-aea7-31d88d380488-openshift-service-ca\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.728912 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.728877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.729075 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.728918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvrh\" (UniqueName: \"kubernetes.io/projected/014799ac-9aae-4f69-aea7-31d88d380488-kube-api-access-dnvrh\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.729075 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.728943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-metrics-certs\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.729075 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.728982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014799ac-9aae-4f69-aea7-31d88d380488-openshift-service-ca\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.729075 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:40:46.729055    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-serving-cert: secret "maas-api-e2e-autores-22fb28-serving-cert" not found
Sep 10 14:40:46.729283 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:40:46.729130    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls podName:014799ac-9aae-4f69-aea7-31d88d380488 nodeName:}" failed. No retries permitted until 2026-09-10 14:40:47.229109917 +0000 UTC m=+773.654847319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls") pod "maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8" (UID: "014799ac-9aae-4f69-aea7-31d88d380488") : secret "maas-api-e2e-autores-22fb28-serving-cert" not found
Sep 10 14:40:46.729622 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.729599    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014799ac-9aae-4f69-aea7-31d88d380488-openshift-service-ca\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.731336 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.731318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-metrics-certs\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:46.736757 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:46.736739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvrh\" (UniqueName: \"kubernetes.io/projected/014799ac-9aae-4f69-aea7-31d88d380488-kube-api-access-dnvrh\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:47.233485 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.233450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:47.235876 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.235851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls\") pod \"maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:47.240141 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.240118    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"]
Sep 10 14:40:47.243535 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.243518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:47.249105 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.249084    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-22fb28-dockercfg-8hlnx\""
Sep 10 14:40:47.249241 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.249225    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-22fb28\""
Sep 10 14:40:47.255638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.255615    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"]
Sep 10 14:40:47.334605 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.334575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5d8f3e-5146-4108-a71d-d98794c0133a-plugins-config-volume\") pod \"payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f\" (UID: \"df5d8f3e-5146-4108-a71d-d98794c0133a\") " pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:47.334767 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.334625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7df\" (UniqueName: \"kubernetes.io/projected/df5d8f3e-5146-4108-a71d-d98794c0133a-kube-api-access-zj7df\") pod \"payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f\" (UID: \"df5d8f3e-5146-4108-a71d-d98794c0133a\") " pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:47.435805 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.435773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7df\" (UniqueName: \"kubernetes.io/projected/df5d8f3e-5146-4108-a71d-d98794c0133a-kube-api-access-zj7df\") pod \"payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f\" (UID: \"df5d8f3e-5146-4108-a71d-d98794c0133a\") " pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:47.435955 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.435855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5d8f3e-5146-4108-a71d-d98794c0133a-plugins-config-volume\") pod \"payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f\" (UID: \"df5d8f3e-5146-4108-a71d-d98794c0133a\") " pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:47.436466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.436448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5d8f3e-5146-4108-a71d-d98794c0133a-plugins-config-volume\") pod \"payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f\" (UID: \"df5d8f3e-5146-4108-a71d-d98794c0133a\") " pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:47.443543 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.443521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7df\" (UniqueName: \"kubernetes.io/projected/df5d8f3e-5146-4108-a71d-d98794c0133a-kube-api-access-zj7df\") pod \"payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f\" (UID: \"df5d8f3e-5146-4108-a71d-d98794c0133a\") " pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:47.451184 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.451163    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:47.553163 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.553119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:47.719502 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.719476    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"]
Sep 10 14:40:47.721373 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:40:47.721349    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014799ac_9aae_4f69_aea7_31d88d380488.slice/crio-b27d304fabf53fb141108dabacbdb87545696a08bc6efcccb16f55954704fa3f WatchSource:0}: Error finding container b27d304fabf53fb141108dabacbdb87545696a08bc6efcccb16f55954704fa3f: Status 404 returned error can't find the container with id b27d304fabf53fb141108dabacbdb87545696a08bc6efcccb16f55954704fa3f
Sep 10 14:40:47.821731 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:47.821626    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"]
Sep 10 14:40:47.824645 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:40:47.824616    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5d8f3e_5146_4108_a71d_d98794c0133a.slice/crio-0c937864bdb5311c58f42115d50ccfa7ca6f4d39c3c4019385f7853ad3b4a057 WatchSource:0}: Error finding container 0c937864bdb5311c58f42115d50ccfa7ca6f4d39c3c4019385f7853ad3b4a057: Status 404 returned error can't find the container with id 0c937864bdb5311c58f42115d50ccfa7ca6f4d39c3c4019385f7853ad3b4a057
Sep 10 14:40:48.341799 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:48.341759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f" event={"ID":"df5d8f3e-5146-4108-a71d-d98794c0133a","Type":"ContainerStarted","Data":"8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b"}
Sep 10 14:40:48.342216 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:48.341820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f" event={"ID":"df5d8f3e-5146-4108-a71d-d98794c0133a","Type":"ContainerStarted","Data":"0c937864bdb5311c58f42115d50ccfa7ca6f4d39c3c4019385f7853ad3b4a057"}
Sep 10 14:40:48.342216 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:48.341862    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:40:48.343009 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:48.342986    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8" event={"ID":"014799ac-9aae-4f69-aea7-31d88d380488","Type":"ContainerStarted","Data":"b27d304fabf53fb141108dabacbdb87545696a08bc6efcccb16f55954704fa3f"}
Sep 10 14:40:48.358645 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:48.358601    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f" podStartSLOduration=1.358586471 podStartE2EDuration="1.358586471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:48.357415948 +0000 UTC m=+774.783153372" watchObservedRunningTime="2026-09-10 14:40:48.358586471 +0000 UTC m=+774.784323890"
Sep 10 14:40:51.357528 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:51.357496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8" event={"ID":"014799ac-9aae-4f69-aea7-31d88d380488","Type":"ContainerStarted","Data":"2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd"}
Sep 10 14:40:51.357953 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:51.357609    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:51.374798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:51.374751    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8" podStartSLOduration=2.7947497610000003 podStartE2EDuration="5.374737665s" totalImagesPullingTime="2.579987904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:40:51.373038957 +0000 UTC m=+777.798776376" watchObservedRunningTime="2026-09-10 14:40:51.374737665 +0000 UTC m=+777.800475084"
Sep 10 14:40:57.368963 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:57.368933    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:40:59.349336 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:40:59.349308    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:41:04.293872 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.293832    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"]
Sep 10 14:41:04.297533 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.297511    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.299608 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.299588    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2dccd5-serving-cert\""
Sep 10 14:41:04.309246 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.309215    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"]
Sep 10 14:41:04.370882 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.370857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3842443f-d383-4466-8d68-4a4755e07e67-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.370984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.370891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvq27\" (UniqueName: \"kubernetes.io/projected/3842443f-d383-4466-8d68-4a4755e07e67-kube-api-access-fvq27\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.370984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.370910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-metrics-certs\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.370984 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.370949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-maas-api-tls\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.471773 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.471743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3842443f-d383-4466-8d68-4a4755e07e67-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.471911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.471779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvq27\" (UniqueName: \"kubernetes.io/projected/3842443f-d383-4466-8d68-4a4755e07e67-kube-api-access-fvq27\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.471911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.471797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-metrics-certs\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.471911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.471828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-maas-api-tls\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.472450 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.472420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3842443f-d383-4466-8d68-4a4755e07e67-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.474208 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.474188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-metrics-certs\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.474295 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.474273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-maas-api-tls\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.479296 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.479276    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvq27\" (UniqueName: \"kubernetes.io/projected/3842443f-d383-4466-8d68-4a4755e07e67-kube-api-access-fvq27\") pod \"maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.608712 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.608635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:04.873526 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.873489    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"]
Sep 10 14:41:04.875653 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:41:04.875620    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3842443f_d383_4466_8d68_4a4755e07e67.slice/crio-f189380527e4b79e3c98f0bb6c7d59e69b0f52313b45c2b5a03d5e3729b5f695 WatchSource:0}: Error finding container f189380527e4b79e3c98f0bb6c7d59e69b0f52313b45c2b5a03d5e3729b5f695: Status 404 returned error can't find the container with id f189380527e4b79e3c98f0bb6c7d59e69b0f52313b45c2b5a03d5e3729b5f695
Sep 10 14:41:04.916218 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.916190    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"]
Sep 10 14:41:04.918975 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.918960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:04.921172 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.921149    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2dccd5\""
Sep 10 14:41:04.930238 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.930212    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"]
Sep 10 14:41:04.954075 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.954053    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"]
Sep 10 14:41:04.955991 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.955977    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:04.958027 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.958009    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2dccd5-dockercfg-r6xgm\""
Sep 10 14:41:04.969209 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:04.969189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"]
Sep 10 14:41:05.076516 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.076483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jhlx\" (UniqueName: \"kubernetes.io/projected/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-kube-api-access-9jhlx\") pod \"payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m\" (UID: \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:05.076678 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.076529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2jt\" (UniqueName: \"kubernetes.io/projected/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-kube-api-access-kp2jt\") pod \"payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk\" (UID: \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:05.076678 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.076645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk\" (UID: \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:05.076903 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.076737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m\" (UID: \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:05.177999 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.177965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m\" (UID: \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:05.178180 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.178015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jhlx\" (UniqueName: \"kubernetes.io/projected/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-kube-api-access-9jhlx\") pod \"payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m\" (UID: \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:05.178180 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.178050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2jt\" (UniqueName: \"kubernetes.io/projected/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-kube-api-access-kp2jt\") pod \"payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk\" (UID: \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:05.178180 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.178121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk\" (UID: \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:05.178839 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.178815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk\" (UID: \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:05.178936 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.178816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m\" (UID: \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:05.186671 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.186651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2jt\" (UniqueName: \"kubernetes.io/projected/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-kube-api-access-kp2jt\") pod \"payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk\" (UID: \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:05.186671 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.186665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jhlx\" (UniqueName: \"kubernetes.io/projected/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-kube-api-access-9jhlx\") pod \"payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m\" (UID: \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:05.230679 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.230650    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:05.265114 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.265088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:05.426242 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.426203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp" event={"ID":"3842443f-d383-4466-8d68-4a4755e07e67","Type":"ContainerStarted","Data":"8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707"}
Sep 10 14:41:05.426665 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.426252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp" event={"ID":"3842443f-d383-4466-8d68-4a4755e07e67","Type":"ContainerStarted","Data":"f189380527e4b79e3c98f0bb6c7d59e69b0f52313b45c2b5a03d5e3729b5f695"}
Sep 10 14:41:05.426665 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.426349    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:05.444348 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.444269    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp" podStartSLOduration=1.056373642 podStartE2EDuration="1.444257813s" totalImagesPullingTime="387.884171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:05.443269709 +0000 UTC m=+791.869007130" watchObservedRunningTime="2026-09-10 14:41:05.444257813 +0000 UTC m=+791.869995232"
Sep 10 14:41:05.500861 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.500837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"]
Sep 10 14:41:05.502975 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:41:05.502946    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbd3ba9_6106_47b5_91f6_20b40e3946e6.slice/crio-227e93e804d0629514d290b0ec1298baabf960967512ce106d37dde2eccc0007 WatchSource:0}: Error finding container 227e93e804d0629514d290b0ec1298baabf960967512ce106d37dde2eccc0007: Status 404 returned error can't find the container with id 227e93e804d0629514d290b0ec1298baabf960967512ce106d37dde2eccc0007
Sep 10 14:41:05.545139 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:05.545114    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"]
Sep 10 14:41:05.555431 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:41:05.555381    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8c3982_245a_4d7f_a23c_64f8ba95d39d.slice/crio-6abf670d65b4df208d98883ad81e373034df89cf96929f9a34dd582ccd21ee10 WatchSource:0}: Error finding container 6abf670d65b4df208d98883ad81e373034df89cf96929f9a34dd582ccd21ee10: Status 404 returned error can't find the container with id 6abf670d65b4df208d98883ad81e373034df89cf96929f9a34dd582ccd21ee10
Sep 10 14:41:06.433205 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:06.433170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m" event={"ID":"bc8c3982-245a-4d7f-a23c-64f8ba95d39d","Type":"ContainerStarted","Data":"9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64"}
Sep 10 14:41:06.433639 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:06.433238    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:41:06.433639 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:06.433261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m" event={"ID":"bc8c3982-245a-4d7f-a23c-64f8ba95d39d","Type":"ContainerStarted","Data":"6abf670d65b4df208d98883ad81e373034df89cf96929f9a34dd582ccd21ee10"}
Sep 10 14:41:06.434833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:06.434799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk" event={"ID":"6dbd3ba9-6106-47b5-91f6-20b40e3946e6","Type":"ContainerStarted","Data":"e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce"}
Sep 10 14:41:06.435000 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:06.434984    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:06.435056 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:06.435021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk" event={"ID":"6dbd3ba9-6106-47b5-91f6-20b40e3946e6","Type":"ContainerStarted","Data":"227e93e804d0629514d290b0ec1298baabf960967512ce106d37dde2eccc0007"}
Sep 10 14:41:06.450406 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:06.450356    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m" podStartSLOduration=2.450339021 podStartE2EDuration="2.450339021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:06.448755455 +0000 UTC m=+792.874492876" watchObservedRunningTime="2026-09-10 14:41:06.450339021 +0000 UTC m=+792.876076442"
Sep 10 14:41:06.465410 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:06.465367    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk" podStartSLOduration=2.4653578879999998 podStartE2EDuration="2.465357888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:06.463226472 +0000 UTC m=+792.888963890" watchObservedRunningTime="2026-09-10 14:41:06.465357888 +0000 UTC m=+792.891095307"
Sep 10 14:41:11.440525 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:11.440498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:41:17.442154 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:17.442073    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:41:17.442154 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:41:17.442151    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:42:25.823778 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.823745    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"]
Sep 10 14:42:25.826103 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.826081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.828186 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.828164    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-22fb28\"/\"kube-root-ca.crt\""
Sep 10 14:42:25.828952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.828924    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-22fb28\"/\"default-dockercfg-9hbxw\""
Sep 10 14:42:25.829091 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.829013    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-22fb28\"/\"autores-model-22fb28-kserve-self-signed-certs\""
Sep 10 14:42:25.829091 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.829013    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-22fb28\"/\"openshift-service-ca.crt\""
Sep 10 14:42:25.837874 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.837853    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"]
Sep 10 14:42:25.842801 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.842746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrnwp\" (UniqueName: \"kubernetes.io/projected/c102f1a3-2621-4f79-9e83-be918b82e741-kube-api-access-zrnwp\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.842974 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.842806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-tmp-dir\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.842974 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.842834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-model-cache\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.842974 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.842870    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c102f1a3-2621-4f79-9e83-be918b82e741-tls-certs\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.842974 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.842891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-dshm\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.842974 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.842953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-home\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944039 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-home\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944225 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrnwp\" (UniqueName: \"kubernetes.io/projected/c102f1a3-2621-4f79-9e83-be918b82e741-kube-api-access-zrnwp\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944225 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-tmp-dir\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944225 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-model-cache\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944225 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c102f1a3-2621-4f79-9e83-be918b82e741-tls-certs\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944455 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-dshm\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-model-cache\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-tmp-dir\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.944817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.944765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-home\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.947286 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.947259    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-dshm\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.947403 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.947307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c102f1a3-2621-4f79-9e83-be918b82e741-tls-certs\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:25.952265 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:25.952243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrnwp\" (UniqueName: \"kubernetes.io/projected/c102f1a3-2621-4f79-9e83-be918b82e741-kube-api-access-zrnwp\") pod \"autores-model-22fb28-kserve-5dcfc9697f-x9vlx\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") " pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:26.136498 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:26.136421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:26.407270 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:26.407245    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"]
Sep 10 14:42:26.409053 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:42:26.409027    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc102f1a3_2621_4f79_9e83_be918b82e741.slice/crio-d839a6e90a5a21d0d7d7ee223887c7ab126c7dd9df9e39ec33ea925d7cb27d36 WatchSource:0}: Error finding container d839a6e90a5a21d0d7d7ee223887c7ab126c7dd9df9e39ec33ea925d7cb27d36: Status 404 returned error can't find the container with id d839a6e90a5a21d0d7d7ee223887c7ab126c7dd9df9e39ec33ea925d7cb27d36
Sep 10 14:42:26.787687 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:26.787597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx" event={"ID":"c102f1a3-2621-4f79-9e83-be918b82e741","Type":"ContainerStarted","Data":"d839a6e90a5a21d0d7d7ee223887c7ab126c7dd9df9e39ec33ea925d7cb27d36"}
Sep 10 14:42:27.793756 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:27.793721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx" event={"ID":"c102f1a3-2621-4f79-9e83-be918b82e741","Type":"ContainerStarted","Data":"09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf"}
Sep 10 14:42:27.808507 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:27.808459    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx" podStartSLOduration=1.8449118960000002 podStartE2EDuration="2.808445123s" totalImagesPullingTime="963.533227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:27.807737129 +0000 UTC m=+874.233474548" watchObservedRunningTime="2026-09-10 14:42:27.808445123 +0000 UTC m=+874.234182541"
Sep 10 14:42:36.137346 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:36.137313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:36.137887 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:36.137359    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:36.151122 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:36.151092    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:36.852345 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:36.852314    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:42:54.023493 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:54.023421    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:42:54.023493 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:54.023421    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:42:54.027742 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:54.027722    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:42:54.028019 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:54.027999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:42:55.270240 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:42:55.270208    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"]
Sep 10 14:43:05.589711 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.589658    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"]
Sep 10 14:43:05.590166 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.589889    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8" podUID="014799ac-9aae-4f69-aea7-31d88d380488" containerName="maas-api" containerID="cri-o://2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd" gracePeriod=30
Sep 10 14:43:05.681620 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.681589    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"]
Sep 10 14:43:05.681841 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.681819    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f" podUID="df5d8f3e-5146-4108-a71d-d98794c0133a" containerName="payload-processing" containerID="cri-o://8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b" gracePeriod=30
Sep 10 14:43:05.849627 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.849523    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:43:05.912210 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.912185    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls\") pod \"014799ac-9aae-4f69-aea7-31d88d380488\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") "
Sep 10 14:43:05.912210 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.912225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014799ac-9aae-4f69-aea7-31d88d380488-openshift-service-ca\") pod \"014799ac-9aae-4f69-aea7-31d88d380488\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") "
Sep 10 14:43:05.912466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.912258    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnvrh\" (UniqueName: \"kubernetes.io/projected/014799ac-9aae-4f69-aea7-31d88d380488-kube-api-access-dnvrh\") pod \"014799ac-9aae-4f69-aea7-31d88d380488\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") "
Sep 10 14:43:05.912466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.912319    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-metrics-certs\") pod \"014799ac-9aae-4f69-aea7-31d88d380488\" (UID: \"014799ac-9aae-4f69-aea7-31d88d380488\") "
Sep 10 14:43:05.912604 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.912574    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014799ac-9aae-4f69-aea7-31d88d380488-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "014799ac-9aae-4f69-aea7-31d88d380488" (UID: "014799ac-9aae-4f69-aea7-31d88d380488"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:43:05.914464 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.914432    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "014799ac-9aae-4f69-aea7-31d88d380488" (UID: "014799ac-9aae-4f69-aea7-31d88d380488"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:05.914570 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.914462    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "014799ac-9aae-4f69-aea7-31d88d380488" (UID: "014799ac-9aae-4f69-aea7-31d88d380488"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:05.914570 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.914470    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014799ac-9aae-4f69-aea7-31d88d380488-kube-api-access-dnvrh" (OuterVolumeSpecName: "kube-api-access-dnvrh") pod "014799ac-9aae-4f69-aea7-31d88d380488" (UID: "014799ac-9aae-4f69-aea7-31d88d380488"). InnerVolumeSpecName "kube-api-access-dnvrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:05.955177 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.955154    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:43:05.976390 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.976354    2566 generic.go:356] "Generic (PLEG): container finished" podID="df5d8f3e-5146-4108-a71d-d98794c0133a" containerID="8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b" exitCode=0
Sep 10 14:43:05.976501 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.976409    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"
Sep 10 14:43:05.976501 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.976449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f" event={"ID":"df5d8f3e-5146-4108-a71d-d98794c0133a","Type":"ContainerDied","Data":"8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b"}
Sep 10 14:43:05.976612 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.976528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f" event={"ID":"df5d8f3e-5146-4108-a71d-d98794c0133a","Type":"ContainerDied","Data":"0c937864bdb5311c58f42115d50ccfa7ca6f4d39c3c4019385f7853ad3b4a057"}
Sep 10 14:43:05.976612 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.976564    2566 scope.go:117] "RemoveContainer" containerID="8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b"
Sep 10 14:43:05.978797 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.978772    2566 generic.go:356] "Generic (PLEG): container finished" podID="014799ac-9aae-4f69-aea7-31d88d380488" containerID="2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd" exitCode=0
Sep 10 14:43:05.978927 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.978864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8" event={"ID":"014799ac-9aae-4f69-aea7-31d88d380488","Type":"ContainerDied","Data":"2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd"}
Sep 10 14:43:05.978927 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.978904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8" event={"ID":"014799ac-9aae-4f69-aea7-31d88d380488","Type":"ContainerDied","Data":"b27d304fabf53fb141108dabacbdb87545696a08bc6efcccb16f55954704fa3f"}
Sep 10 14:43:05.979085 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.978968    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"
Sep 10 14:43:05.987241 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.987220    2566 scope.go:117] "RemoveContainer" containerID="8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b"
Sep 10 14:43:05.987543 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:43:05.987523    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b\": container with ID starting with 8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b not found: ID does not exist" containerID="8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b"
Sep 10 14:43:05.987594 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.987553    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b"} err="failed to get container status \"8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b\": rpc error: code = NotFound desc = could not find container \"8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b\": container with ID starting with 8387c82c41ccb4f2056728dbae3b51a7406977be2287172764dff647ad9d7b7b not found: ID does not exist"
Sep 10 14:43:05.987594 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.987571    2566 scope.go:117] "RemoveContainer" containerID="2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd"
Sep 10 14:43:05.995581 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.995565    2566 scope.go:117] "RemoveContainer" containerID="2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd"
Sep 10 14:43:05.995914 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:43:05.995896    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd\": container with ID starting with 2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd not found: ID does not exist" containerID="2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd"
Sep 10 14:43:05.995973 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:05.995922    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd"} err="failed to get container status \"2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd\": rpc error: code = NotFound desc = could not find container \"2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd\": container with ID starting with 2bc111cb8a041536f9f52df68577de4d20f69eb6b719920b73db6c7d9c9503dd not found: ID does not exist"
Sep 10 14:43:06.013421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.013397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj7df\" (UniqueName: \"kubernetes.io/projected/df5d8f3e-5146-4108-a71d-d98794c0133a-kube-api-access-zj7df\") pod \"df5d8f3e-5146-4108-a71d-d98794c0133a\" (UID: \"df5d8f3e-5146-4108-a71d-d98794c0133a\") "
Sep 10 14:43:06.013542 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.013514    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5d8f3e-5146-4108-a71d-d98794c0133a-plugins-config-volume\") pod \"df5d8f3e-5146-4108-a71d-d98794c0133a\" (UID: \"df5d8f3e-5146-4108-a71d-d98794c0133a\") "
Sep 10 14:43:06.014011 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.013762    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:06.014011 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.013792    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014799ac-9aae-4f69-aea7-31d88d380488-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:06.014011 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.013807    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvrh\" (UniqueName: \"kubernetes.io/projected/014799ac-9aae-4f69-aea7-31d88d380488-kube-api-access-dnvrh\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:06.014011 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.013820    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014799ac-9aae-4f69-aea7-31d88d380488-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:06.014011 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.013933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5d8f3e-5146-4108-a71d-d98794c0133a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df5d8f3e-5146-4108-a71d-d98794c0133a" (UID: "df5d8f3e-5146-4108-a71d-d98794c0133a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:43:06.016352 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.016328    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5d8f3e-5146-4108-a71d-d98794c0133a-kube-api-access-zj7df" (OuterVolumeSpecName: "kube-api-access-zj7df") pod "df5d8f3e-5146-4108-a71d-d98794c0133a" (UID: "df5d8f3e-5146-4108-a71d-d98794c0133a"). InnerVolumeSpecName "kube-api-access-zj7df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:06.016656 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.016638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"]
Sep 10 14:43:06.023911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.023889    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-22fb28-c8d996bc5-nnpw8"]
Sep 10 14:43:06.110531 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.110456    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014799ac-9aae-4f69-aea7-31d88d380488" path="/var/lib/kubelet/pods/014799ac-9aae-4f69-aea7-31d88d380488/volumes"
Sep 10 14:43:06.114181 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.114154    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7df\" (UniqueName: \"kubernetes.io/projected/df5d8f3e-5146-4108-a71d-d98794c0133a-kube-api-access-zj7df\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:06.114181 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.114178    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5d8f3e-5146-4108-a71d-d98794c0133a-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:06.306023 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.305996    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"]
Sep 10 14:43:06.318089 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:06.318062    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-22fb28-5d6b58f6df-2q66f"]
Sep 10 14:43:08.110483 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:08.110447    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5d8f3e-5146-4108-a71d-d98794c0133a" path="/var/lib/kubelet/pods/df5d8f3e-5146-4108-a71d-d98794c0133a/volumes"
Sep 10 14:43:10.289382 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.289313    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx" podUID="c102f1a3-2621-4f79-9e83-be918b82e741" containerName="main" containerID="cri-o://09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf" gracePeriod=45
Sep 10 14:43:10.540270 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.540216    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:43:10.648751 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.648712    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-dshm\") pod \"c102f1a3-2621-4f79-9e83-be918b82e741\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") "
Sep 10 14:43:10.648916 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.648762    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-tmp-dir\") pod \"c102f1a3-2621-4f79-9e83-be918b82e741\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") "
Sep 10 14:43:10.648916 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.648783    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-home\") pod \"c102f1a3-2621-4f79-9e83-be918b82e741\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") "
Sep 10 14:43:10.648916 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.648828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrnwp\" (UniqueName: \"kubernetes.io/projected/c102f1a3-2621-4f79-9e83-be918b82e741-kube-api-access-zrnwp\") pod \"c102f1a3-2621-4f79-9e83-be918b82e741\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") "
Sep 10 14:43:10.648916 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.648858    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-model-cache\") pod \"c102f1a3-2621-4f79-9e83-be918b82e741\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") "
Sep 10 14:43:10.649140 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.649022    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c102f1a3-2621-4f79-9e83-be918b82e741-tls-certs\") pod \"c102f1a3-2621-4f79-9e83-be918b82e741\" (UID: \"c102f1a3-2621-4f79-9e83-be918b82e741\") "
Sep 10 14:43:10.649140 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.649039    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c102f1a3-2621-4f79-9e83-be918b82e741" (UID: "c102f1a3-2621-4f79-9e83-be918b82e741"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:10.649140 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.649079    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-home" (OuterVolumeSpecName: "home") pod "c102f1a3-2621-4f79-9e83-be918b82e741" (UID: "c102f1a3-2621-4f79-9e83-be918b82e741"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:10.649140 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.649100    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-model-cache" (OuterVolumeSpecName: "model-cache") pod "c102f1a3-2621-4f79-9e83-be918b82e741" (UID: "c102f1a3-2621-4f79-9e83-be918b82e741"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:10.649295 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.649279    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:10.649332 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.649298    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:10.649332 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.649306    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:10.651069 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.651036    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c102f1a3-2621-4f79-9e83-be918b82e741-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c102f1a3-2621-4f79-9e83-be918b82e741" (UID: "c102f1a3-2621-4f79-9e83-be918b82e741"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:43:10.651177 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.651095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-dshm" (OuterVolumeSpecName: "dshm") pod "c102f1a3-2621-4f79-9e83-be918b82e741" (UID: "c102f1a3-2621-4f79-9e83-be918b82e741"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:43:10.651177 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.651118    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c102f1a3-2621-4f79-9e83-be918b82e741-kube-api-access-zrnwp" (OuterVolumeSpecName: "kube-api-access-zrnwp") pod "c102f1a3-2621-4f79-9e83-be918b82e741" (UID: "c102f1a3-2621-4f79-9e83-be918b82e741"). InnerVolumeSpecName "kube-api-access-zrnwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:43:10.749897 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.749861    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrnwp\" (UniqueName: \"kubernetes.io/projected/c102f1a3-2621-4f79-9e83-be918b82e741-kube-api-access-zrnwp\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:10.749897 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.749892    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c102f1a3-2621-4f79-9e83-be918b82e741-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:10.749897 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:10.749903    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c102f1a3-2621-4f79-9e83-be918b82e741-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:43:11.003841 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.003804    2566 generic.go:356] "Generic (PLEG): container finished" podID="c102f1a3-2621-4f79-9e83-be918b82e741" containerID="09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf" exitCode=0
Sep 10 14:43:11.003841 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.003847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx" event={"ID":"c102f1a3-2621-4f79-9e83-be918b82e741","Type":"ContainerDied","Data":"09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf"}
Sep 10 14:43:11.004049 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.003849    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"
Sep 10 14:43:11.004049 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.003883    2566 scope.go:117] "RemoveContainer" containerID="09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf"
Sep 10 14:43:11.004049 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.003873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx" event={"ID":"c102f1a3-2621-4f79-9e83-be918b82e741","Type":"ContainerDied","Data":"d839a6e90a5a21d0d7d7ee223887c7ab126c7dd9df9e39ec33ea925d7cb27d36"}
Sep 10 14:43:11.012767 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.012749    2566 scope.go:117] "RemoveContainer" containerID="09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf"
Sep 10 14:43:11.013038 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:43:11.013020    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf\": container with ID starting with 09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf not found: ID does not exist" containerID="09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf"
Sep 10 14:43:11.013103 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.013045    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf"} err="failed to get container status \"09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf\": rpc error: code = NotFound desc = could not find container \"09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf\": container with ID starting with 09e2f9e34a0c832dd6cf83ad4bbf7ae6a8d5ddb23877dc505faec49dd9e2b2cf not found: ID does not exist"
Sep 10 14:43:11.028414 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.028389    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"]
Sep 10 14:43:11.035639 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:11.035619    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-22fb28/autores-model-22fb28-kserve-5dcfc9697f-x9vlx"]
Sep 10 14:43:12.110666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:12.110631    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c102f1a3-2621-4f79-9e83-be918b82e741" path="/var/lib/kubelet/pods/c102f1a3-2621-4f79-9e83-be918b82e741/volumes"
Sep 10 14:43:19.900740 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.900683    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"]
Sep 10 14:43:19.901099 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901031    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014799ac-9aae-4f69-aea7-31d88d380488" containerName="maas-api"
Sep 10 14:43:19.901099 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901043    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="014799ac-9aae-4f69-aea7-31d88d380488" containerName="maas-api"
Sep 10 14:43:19.901099 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901060    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c102f1a3-2621-4f79-9e83-be918b82e741" containerName="main"
Sep 10 14:43:19.901099 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901066    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c102f1a3-2621-4f79-9e83-be918b82e741" containerName="main"
Sep 10 14:43:19.901099 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901081    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5d8f3e-5146-4108-a71d-d98794c0133a" containerName="payload-processing"
Sep 10 14:43:19.901099 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901087    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5d8f3e-5146-4108-a71d-d98794c0133a" containerName="payload-processing"
Sep 10 14:43:19.901285 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901137    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5d8f3e-5146-4108-a71d-d98794c0133a" containerName="payload-processing"
Sep 10 14:43:19.901285 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901144    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c102f1a3-2621-4f79-9e83-be918b82e741" containerName="main"
Sep 10 14:43:19.901285 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.901150    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="014799ac-9aae-4f69-aea7-31d88d380488" containerName="maas-api"
Sep 10 14:43:19.905339 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.905318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:19.907805 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.907780    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c072e8-serving-cert\""
Sep 10 14:43:19.949881 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:19.949855    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"]
Sep 10 14:43:20.023077 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.023045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-maas-api-tls\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.023077 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.023079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-metrics-certs\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.023275 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.023104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h525p\" (UniqueName: \"kubernetes.io/projected/1d5c2706-e582-4a25-b807-2d37082bb123-kube-api-access-h525p\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.023275 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.023171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c2706-e582-4a25-b807-2d37082bb123-openshift-service-ca\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.124028 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.123995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-maas-api-tls\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.124028 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.124036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-metrics-certs\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.124254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.124070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h525p\" (UniqueName: \"kubernetes.io/projected/1d5c2706-e582-4a25-b807-2d37082bb123-kube-api-access-h525p\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.124254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.124099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c2706-e582-4a25-b807-2d37082bb123-openshift-service-ca\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.124820 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.124792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c2706-e582-4a25-b807-2d37082bb123-openshift-service-ca\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.126664 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.126622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-metrics-certs\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.126843 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.126816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-maas-api-tls\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.141737 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.141711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h525p\" (UniqueName: \"kubernetes.io/projected/1d5c2706-e582-4a25-b807-2d37082bb123-kube-api-access-h525p\") pod \"maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.215628 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.215568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:20.494608 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:20.494584    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"]
Sep 10 14:43:20.496546 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:43:20.496512    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5c2706_e582_4a25_b807_2d37082bb123.slice/crio-d8208b53b27b95656ca2742271456e2cd5c2193e947f5b75250f49a05efae79c WatchSource:0}: Error finding container d8208b53b27b95656ca2742271456e2cd5c2193e947f5b75250f49a05efae79c: Status 404 returned error can't find the container with id d8208b53b27b95656ca2742271456e2cd5c2193e947f5b75250f49a05efae79c
Sep 10 14:43:21.059687 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.059655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb" event={"ID":"1d5c2706-e582-4a25-b807-2d37082bb123","Type":"ContainerStarted","Data":"d8208b53b27b95656ca2742271456e2cd5c2193e947f5b75250f49a05efae79c"}
Sep 10 14:43:21.101288 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.101256    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"]
Sep 10 14:43:21.104520 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.104503    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:21.107865 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.107843    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c072e8-dockercfg-lttn7\""
Sep 10 14:43:21.108149 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.108128    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c072e8\""
Sep 10 14:43:21.177330 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.177299    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"]
Sep 10 14:43:21.234703 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.234670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8hv8\" (UniqueName: \"kubernetes.io/projected/e2acdbaf-35c1-46e1-931a-7d02794d431f-kube-api-access-r8hv8\") pod \"payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8\" (UID: \"e2acdbaf-35c1-46e1-931a-7d02794d431f\") " pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:21.234853 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.234730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2acdbaf-35c1-46e1-931a-7d02794d431f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8\" (UID: \"e2acdbaf-35c1-46e1-931a-7d02794d431f\") " pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:21.335548 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.335470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8hv8\" (UniqueName: \"kubernetes.io/projected/e2acdbaf-35c1-46e1-931a-7d02794d431f-kube-api-access-r8hv8\") pod \"payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8\" (UID: \"e2acdbaf-35c1-46e1-931a-7d02794d431f\") " pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:21.335548 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.335517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2acdbaf-35c1-46e1-931a-7d02794d431f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8\" (UID: \"e2acdbaf-35c1-46e1-931a-7d02794d431f\") " pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:21.336240 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.336220    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2acdbaf-35c1-46e1-931a-7d02794d431f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8\" (UID: \"e2acdbaf-35c1-46e1-931a-7d02794d431f\") " pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:21.348968 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.348935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8hv8\" (UniqueName: \"kubernetes.io/projected/e2acdbaf-35c1-46e1-931a-7d02794d431f-kube-api-access-r8hv8\") pod \"payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8\" (UID: \"e2acdbaf-35c1-46e1-931a-7d02794d431f\") " pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:21.415550 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.415508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:21.726216 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:21.726181    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"]
Sep 10 14:43:21.729245 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:43:21.729219    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2acdbaf_35c1_46e1_931a_7d02794d431f.slice/crio-7dd42b3cae7adf645faa6da0836b0aeaed03bb076a4c663fde6bd2e4fd173f50 WatchSource:0}: Error finding container 7dd42b3cae7adf645faa6da0836b0aeaed03bb076a4c663fde6bd2e4fd173f50: Status 404 returned error can't find the container with id 7dd42b3cae7adf645faa6da0836b0aeaed03bb076a4c663fde6bd2e4fd173f50
Sep 10 14:43:22.066048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:22.066013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb" event={"ID":"1d5c2706-e582-4a25-b807-2d37082bb123","Type":"ContainerStarted","Data":"36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797"}
Sep 10 14:43:22.066503 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:22.066085    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:22.067610 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:22.067588    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8" event={"ID":"e2acdbaf-35c1-46e1-931a-7d02794d431f","Type":"ContainerStarted","Data":"35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7"}
Sep 10 14:43:22.067676 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:22.067628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8" event={"ID":"e2acdbaf-35c1-46e1-931a-7d02794d431f","Type":"ContainerStarted","Data":"7dd42b3cae7adf645faa6da0836b0aeaed03bb076a4c663fde6bd2e4fd173f50"}
Sep 10 14:43:22.067676 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:22.067648    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:43:22.112851 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:22.112799    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb" podStartSLOduration=2.078009774 podStartE2EDuration="3.112783248s" totalImagesPullingTime="1.034773474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:22.111047515 +0000 UTC m=+928.536784934" watchObservedRunningTime="2026-09-10 14:43:22.112783248 +0000 UTC m=+928.538520667"
Sep 10 14:43:22.146281 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:22.146230    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8" podStartSLOduration=1.146215582 podStartE2EDuration="1.146215582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:22.143763601 +0000 UTC m=+928.569501019" watchObservedRunningTime="2026-09-10 14:43:22.146215582 +0000 UTC m=+928.571953049"
Sep 10 14:43:28.078245 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:28.078217    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:43:33.074520 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:43:33.074485    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:44:04.727448 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.727415    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"]
Sep 10 14:44:04.732356 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.732331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.736722 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.736667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c072e8\"/\"openshift-service-ca.crt\""
Sep 10 14:44:04.737494 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.737471    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c072e8\"/\"kube-root-ca.crt\""
Sep 10 14:44:04.737602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.737553    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c072e8\"/\"explicit-model-c072e8-kserve-self-signed-certs\""
Sep 10 14:44:04.737602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.737562    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c072e8\"/\"default-dockercfg-z9nvx\""
Sep 10 14:44:04.745887 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.745863    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"]
Sep 10 14:44:04.782035 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.782005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe3397-6d93-40bc-9f89-ba17d88baee3-tls-certs\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.782194 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.782042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-model-cache\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.782194 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.782066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-dshm\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.782194 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.782148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-tmp-dir\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.782194 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.782192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-home\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.782354 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.782272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4kgd\" (UniqueName: \"kubernetes.io/projected/60fe3397-6d93-40bc-9f89-ba17d88baee3-kube-api-access-p4kgd\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.883071 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kgd\" (UniqueName: \"kubernetes.io/projected/60fe3397-6d93-40bc-9f89-ba17d88baee3-kube-api-access-p4kgd\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.883239 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe3397-6d93-40bc-9f89-ba17d88baee3-tls-certs\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.883239 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-model-cache\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.883239 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-dshm\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.883239 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-tmp-dir\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.883239 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-home\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.883688 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-home\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.883887 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-tmp-dir\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.884049 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.883809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-model-cache\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.886395 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.886349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-dshm\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.887287 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.887255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe3397-6d93-40bc-9f89-ba17d88baee3-tls-certs\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:04.891928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:04.891904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kgd\" (UniqueName: \"kubernetes.io/projected/60fe3397-6d93-40bc-9f89-ba17d88baee3-kube-api-access-p4kgd\") pod \"explicit-model-c072e8-kserve-688857695f-5qrzw\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") " pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:05.046312 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:05.046231    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:05.316266 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:05.316237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"]
Sep 10 14:44:05.318173 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:44:05.318149    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60fe3397_6d93_40bc_9f89_ba17d88baee3.slice/crio-7be64350a52138f15b29dc03fc095c8e6b3f0a3c3fdbaeb02f4de0696e1019b6 WatchSource:0}: Error finding container 7be64350a52138f15b29dc03fc095c8e6b3f0a3c3fdbaeb02f4de0696e1019b6: Status 404 returned error can't find the container with id 7be64350a52138f15b29dc03fc095c8e6b3f0a3c3fdbaeb02f4de0696e1019b6
Sep 10 14:44:06.270602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:06.270566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw" event={"ID":"60fe3397-6d93-40bc-9f89-ba17d88baee3","Type":"ContainerStarted","Data":"f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7"}
Sep 10 14:44:06.270993 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:06.270611    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw" event={"ID":"60fe3397-6d93-40bc-9f89-ba17d88baee3","Type":"ContainerStarted","Data":"7be64350a52138f15b29dc03fc095c8e6b3f0a3c3fdbaeb02f4de0696e1019b6"}
Sep 10 14:44:06.287492 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:06.287445    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw" podStartSLOduration=2.287429953 podStartE2EDuration="2.287429953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:06.286382673 +0000 UTC m=+972.712120107" watchObservedRunningTime="2026-09-10 14:44:06.287429953 +0000 UTC m=+972.713167371"
Sep 10 14:44:15.046886 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:15.046849    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:15.047319 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:15.046920    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:15.059788 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:15.059767    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:15.320799 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:15.320661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:44:50.699914 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:50.699877    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"]
Sep 10 14:44:50.700798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:50.700755    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb" podUID="1d5c2706-e582-4a25-b807-2d37082bb123" containerName="maas-api" containerID="cri-o://36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797" gracePeriod=30
Sep 10 14:44:50.764265 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:50.764233    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"]
Sep 10 14:44:50.764481 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:50.764440    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8" podUID="e2acdbaf-35c1-46e1-931a-7d02794d431f" containerName="payload-processing" containerID="cri-o://35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7" gracePeriod=30
Sep 10 14:44:51.015653 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.015625    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:44:51.044741 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.044716    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:44:51.173023 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.172985    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h525p\" (UniqueName: \"kubernetes.io/projected/1d5c2706-e582-4a25-b807-2d37082bb123-kube-api-access-h525p\") pod \"1d5c2706-e582-4a25-b807-2d37082bb123\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") "
Sep 10 14:44:51.173183 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2acdbaf-35c1-46e1-931a-7d02794d431f-plugins-config-volume\") pod \"e2acdbaf-35c1-46e1-931a-7d02794d431f\" (UID: \"e2acdbaf-35c1-46e1-931a-7d02794d431f\") "
Sep 10 14:44:51.173183 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173079    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c2706-e582-4a25-b807-2d37082bb123-openshift-service-ca\") pod \"1d5c2706-e582-4a25-b807-2d37082bb123\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") "
Sep 10 14:44:51.173272 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173178    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8hv8\" (UniqueName: \"kubernetes.io/projected/e2acdbaf-35c1-46e1-931a-7d02794d431f-kube-api-access-r8hv8\") pod \"e2acdbaf-35c1-46e1-931a-7d02794d431f\" (UID: \"e2acdbaf-35c1-46e1-931a-7d02794d431f\") "
Sep 10 14:44:51.173272 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173233    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-metrics-certs\") pod \"1d5c2706-e582-4a25-b807-2d37082bb123\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") "
Sep 10 14:44:51.173356 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173280    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-maas-api-tls\") pod \"1d5c2706-e582-4a25-b807-2d37082bb123\" (UID: \"1d5c2706-e582-4a25-b807-2d37082bb123\") "
Sep 10 14:44:51.173413 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173348    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5c2706-e582-4a25-b807-2d37082bb123-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d5c2706-e582-4a25-b807-2d37082bb123" (UID: "1d5c2706-e582-4a25-b807-2d37082bb123"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:44:51.173473 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2acdbaf-35c1-46e1-931a-7d02794d431f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2acdbaf-35c1-46e1-931a-7d02794d431f" (UID: "e2acdbaf-35c1-46e1-931a-7d02794d431f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:44:51.173582 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173564    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2acdbaf-35c1-46e1-931a-7d02794d431f-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:44:51.173631 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.173582    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5c2706-e582-4a25-b807-2d37082bb123-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:44:51.175274 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.175242    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2acdbaf-35c1-46e1-931a-7d02794d431f-kube-api-access-r8hv8" (OuterVolumeSpecName: "kube-api-access-r8hv8") pod "e2acdbaf-35c1-46e1-931a-7d02794d431f" (UID: "e2acdbaf-35c1-46e1-931a-7d02794d431f"). InnerVolumeSpecName "kube-api-access-r8hv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:44:51.175384 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.175311    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d5c2706-e582-4a25-b807-2d37082bb123" (UID: "1d5c2706-e582-4a25-b807-2d37082bb123"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:44:51.175384 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.175338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1d5c2706-e582-4a25-b807-2d37082bb123" (UID: "1d5c2706-e582-4a25-b807-2d37082bb123"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:44:51.175384 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.175366    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5c2706-e582-4a25-b807-2d37082bb123-kube-api-access-h525p" (OuterVolumeSpecName: "kube-api-access-h525p") pod "1d5c2706-e582-4a25-b807-2d37082bb123" (UID: "1d5c2706-e582-4a25-b807-2d37082bb123"). InnerVolumeSpecName "kube-api-access-h525p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:44:51.274991 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.274907    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8hv8\" (UniqueName: \"kubernetes.io/projected/e2acdbaf-35c1-46e1-931a-7d02794d431f-kube-api-access-r8hv8\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:44:51.274991 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.274940    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:44:51.274991 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.274954    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d5c2706-e582-4a25-b807-2d37082bb123-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:44:51.274991 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.274967    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h525p\" (UniqueName: \"kubernetes.io/projected/1d5c2706-e582-4a25-b807-2d37082bb123-kube-api-access-h525p\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:44:51.474245 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.474212    2566 generic.go:356] "Generic (PLEG): container finished" podID="1d5c2706-e582-4a25-b807-2d37082bb123" containerID="36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797" exitCode=0
Sep 10 14:44:51.474430 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.474257    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"
Sep 10 14:44:51.474430 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.474291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb" event={"ID":"1d5c2706-e582-4a25-b807-2d37082bb123","Type":"ContainerDied","Data":"36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797"}
Sep 10 14:44:51.474430 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.474366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb" event={"ID":"1d5c2706-e582-4a25-b807-2d37082bb123","Type":"ContainerDied","Data":"d8208b53b27b95656ca2742271456e2cd5c2193e947f5b75250f49a05efae79c"}
Sep 10 14:44:51.474430 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.474397    2566 scope.go:117] "RemoveContainer" containerID="36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797"
Sep 10 14:44:51.475776 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.475731    2566 generic.go:356] "Generic (PLEG): container finished" podID="e2acdbaf-35c1-46e1-931a-7d02794d431f" containerID="35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7" exitCode=0
Sep 10 14:44:51.475844 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.475789    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"
Sep 10 14:44:51.475844 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.475805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8" event={"ID":"e2acdbaf-35c1-46e1-931a-7d02794d431f","Type":"ContainerDied","Data":"35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7"}
Sep 10 14:44:51.475955 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.475856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8" event={"ID":"e2acdbaf-35c1-46e1-931a-7d02794d431f","Type":"ContainerDied","Data":"7dd42b3cae7adf645faa6da0836b0aeaed03bb076a4c663fde6bd2e4fd173f50"}
Sep 10 14:44:51.485157 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.485138    2566 scope.go:117] "RemoveContainer" containerID="36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797"
Sep 10 14:44:51.485589 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:44:51.485547    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797\": container with ID starting with 36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797 not found: ID does not exist" containerID="36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797"
Sep 10 14:44:51.485729 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.485603    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797"} err="failed to get container status \"36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797\": rpc error: code = NotFound desc = could not find container \"36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797\": container with ID starting with 36b33107f0e50540ec5eadb1df6a373e18fd8e159db7fdebfaa32a945060c797 not found: ID does not exist"
Sep 10 14:44:51.485729 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.485630    2566 scope.go:117] "RemoveContainer" containerID="35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7"
Sep 10 14:44:51.493784 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.493764    2566 scope.go:117] "RemoveContainer" containerID="35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7"
Sep 10 14:44:51.494048 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:44:51.494031    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7\": container with ID starting with 35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7 not found: ID does not exist" containerID="35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7"
Sep 10 14:44:51.494096 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.494055    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7"} err="failed to get container status \"35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7\": rpc error: code = NotFound desc = could not find container \"35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7\": container with ID starting with 35bef7e16a1095483583e2baaca7166e4a418a2a4cb03051ab4fd8137e7258e7 not found: ID does not exist"
Sep 10 14:44:51.501048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.501025    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"]
Sep 10 14:44:51.506291 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.506271    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c072e8-59c8754cd9-9bnlb"]
Sep 10 14:44:51.515656 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.515636    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"]
Sep 10 14:44:51.520396 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:51.520376    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c072e8-94b9ddc8f-vzfx8"]
Sep 10 14:44:52.110432 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.110356    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5c2706-e582-4a25-b807-2d37082bb123" path="/var/lib/kubelet/pods/1d5c2706-e582-4a25-b807-2d37082bb123/volumes"
Sep 10 14:44:52.110787 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.110735    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2acdbaf-35c1-46e1-931a-7d02794d431f" path="/var/lib/kubelet/pods/e2acdbaf-35c1-46e1-931a-7d02794d431f/volumes"
Sep 10 14:44:52.415494 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.415458    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"]
Sep 10 14:44:52.415817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.415805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2acdbaf-35c1-46e1-931a-7d02794d431f" containerName="payload-processing"
Sep 10 14:44:52.415867 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.415819    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2acdbaf-35c1-46e1-931a-7d02794d431f" containerName="payload-processing"
Sep 10 14:44:52.415867 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.415839    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5c2706-e582-4a25-b807-2d37082bb123" containerName="maas-api"
Sep 10 14:44:52.415867 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.415844    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5c2706-e582-4a25-b807-2d37082bb123" containerName="maas-api"
Sep 10 14:44:52.415960 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.415911    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2acdbaf-35c1-46e1-931a-7d02794d431f" containerName="payload-processing"
Sep 10 14:44:52.415960 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.415920    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5c2706-e582-4a25-b807-2d37082bb123" containerName="maas-api"
Sep 10 14:44:52.419960 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.419943    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.422104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.422083    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-785829b5-serving-cert\""
Sep 10 14:44:52.428789 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.428764    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"]
Sep 10 14:44:52.586671 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.586635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.586671 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.586668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-metrics-certs\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.586909 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.586724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g546d\" (UniqueName: \"kubernetes.io/projected/54d835be-3716-468f-9f76-58ce2b538d86-kube-api-access-g546d\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.586909 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.586772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d835be-3716-468f-9f76-58ce2b538d86-openshift-service-ca\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.687752 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.687655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.687752 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.687708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-metrics-certs\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.687752 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.687736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g546d\" (UniqueName: \"kubernetes.io/projected/54d835be-3716-468f-9f76-58ce2b538d86-kube-api-access-g546d\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.687975 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.687763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d835be-3716-468f-9f76-58ce2b538d86-openshift-service-ca\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.687975 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:44:52.687820    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-serving-cert: secret "maas-api-e2e-mt-785829b5-serving-cert" not found
Sep 10 14:44:52.687975 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:44:52.687904    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls podName:54d835be-3716-468f-9f76-58ce2b538d86 nodeName:}" failed. No retries permitted until 2026-09-10 14:44:53.187881377 +0000 UTC m=+1019.613618776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls") pod "maas-api-e2e-mt-785829b5-6fcd69986c-npdpz" (UID: "54d835be-3716-468f-9f76-58ce2b538d86") : secret "maas-api-e2e-mt-785829b5-serving-cert" not found
Sep 10 14:44:52.688460 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.688436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d835be-3716-468f-9f76-58ce2b538d86-openshift-service-ca\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.690188 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.690165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-metrics-certs\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:52.695457 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:52.695435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g546d\" (UniqueName: \"kubernetes.io/projected/54d835be-3716-468f-9f76-58ce2b538d86-kube-api-access-g546d\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:53.191215 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.191177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:53.193572 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.193551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls\") pod \"maas-api-e2e-mt-785829b5-6fcd69986c-npdpz\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:53.332456 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.332422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:53.420536 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.420503    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"]
Sep 10 14:44:53.423926 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.423906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.425997 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.425978    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"openshift-service-ca.crt\""
Sep 10 14:44:53.426127 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.426008    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"kube-root-ca.crt\""
Sep 10 14:44:53.426904 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.426879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"default-dockercfg-r4d7f\""
Sep 10 14:44:53.427002 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.426952    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"distinct-w4-5b0323-kserve-self-signed-certs\""
Sep 10 14:44:53.433156 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.433133    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"]
Sep 10 14:44:53.493839 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.493761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-home\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.493839 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.493797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tls-certs\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.493839 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.493822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5xr\" (UniqueName: \"kubernetes.io/projected/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-kube-api-access-mc5xr\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.494045 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.493895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tmp-dir\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.494045 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.493954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-dshm\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.494045 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.493997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-model-cache\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595109 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-home\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595109 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tls-certs\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595338 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5xr\" (UniqueName: \"kubernetes.io/projected/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-kube-api-access-mc5xr\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595338 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tmp-dir\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595338 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-dshm\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595520 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-model-cache\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595520 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-home\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595613 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tmp-dir\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.595648 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.595633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-model-cache\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.597443 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.597414    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-dshm\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.597576 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.597558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tls-certs\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.603317 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.603292    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"]
Sep 10 14:44:53.604415 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.604371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5xr\" (UniqueName: \"kubernetes.io/projected/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-kube-api-access-mc5xr\") pod \"distinct-w4-5b0323-kserve-7d8846bf94-4bmw5\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") " pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:53.606116 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:44:53.606076    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d835be_3716_468f_9f76_58ce2b538d86.slice/crio-c868000674f54950f515a3f29d89a0be2fe7e8b82d024a31daf9237b021f7351 WatchSource:0}: Error finding container c868000674f54950f515a3f29d89a0be2fe7e8b82d024a31daf9237b021f7351: Status 404 returned error can't find the container with id c868000674f54950f515a3f29d89a0be2fe7e8b82d024a31daf9237b021f7351
Sep 10 14:44:53.734764 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:53.734731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:44:54.007492 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:54.007469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"]
Sep 10 14:44:54.010269 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:44:54.010223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod958d61c1_0ac9_4e18_a9a2_10d2f909ecb3.slice/crio-406e33d71944d43e7d89282eac5929019b61a5d8f0d553026f79d34f49c46e8c WatchSource:0}: Error finding container 406e33d71944d43e7d89282eac5929019b61a5d8f0d553026f79d34f49c46e8c: Status 404 returned error can't find the container with id 406e33d71944d43e7d89282eac5929019b61a5d8f0d553026f79d34f49c46e8c
Sep 10 14:44:54.492056 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:54.492015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5" event={"ID":"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3","Type":"ContainerStarted","Data":"cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b"}
Sep 10 14:44:54.492056 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:54.492062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5" event={"ID":"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3","Type":"ContainerStarted","Data":"406e33d71944d43e7d89282eac5929019b61a5d8f0d553026f79d34f49c46e8c"}
Sep 10 14:44:54.493632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:54.493604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz" event={"ID":"54d835be-3716-468f-9f76-58ce2b538d86","Type":"ContainerStarted","Data":"0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065"}
Sep 10 14:44:54.493787 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:54.493652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz" event={"ID":"54d835be-3716-468f-9f76-58ce2b538d86","Type":"ContainerStarted","Data":"c868000674f54950f515a3f29d89a0be2fe7e8b82d024a31daf9237b021f7351"}
Sep 10 14:44:54.493787 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:54.493727    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:44:54.509129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:54.509087    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5" podStartSLOduration=1.509074319 podStartE2EDuration="1.509074319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:54.507420917 +0000 UTC m=+1020.933158351" watchObservedRunningTime="2026-09-10 14:44:54.509074319 +0000 UTC m=+1020.934811740"
Sep 10 14:44:54.525331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:54.525286    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz" podStartSLOduration=1.934406692 podStartE2EDuration="2.525271966s" totalImagesPullingTime="590.865274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:54.523686761 +0000 UTC m=+1020.949424187" watchObservedRunningTime="2026-09-10 14:44:54.525271966 +0000 UTC m=+1020.951009429"
Sep 10 14:44:56.306964 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:44:56.306926    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"]
Sep 10 14:45:00.146294 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.146262    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj"]
Sep 10 14:45:00.149784 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.149767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj"
Sep 10 14:45:00.152392 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.152366    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gd94c\""
Sep 10 14:45:00.159343 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.159317    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj"]
Sep 10 14:45:00.162619 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.162599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj"
Sep 10 14:45:00.162742 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.162622    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"]
Sep 10 14:45:00.166054 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.166036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"
Sep 10 14:45:00.177082 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.176837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"]
Sep 10 14:45:00.191679 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.191649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"
Sep 10 14:45:00.440330 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.440306    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj"]
Sep 10 14:45:00.442462 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:45:00.442438    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf124b00_7259_4e7d_aa67_0b6072adcbd6.slice/crio-f8efd33821fe539e201294b5a1c33a39dad7741557e3c79accce4b6f8a4d217a WatchSource:0}: Error finding container f8efd33821fe539e201294b5a1c33a39dad7741557e3c79accce4b6f8a4d217a: Status 404 returned error can't find the container with id f8efd33821fe539e201294b5a1c33a39dad7741557e3c79accce4b6f8a4d217a
Sep 10 14:45:00.473936 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.473912    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"]
Sep 10 14:45:00.476159 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:45:00.476139    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f75aad_cd34_4d42_9aad_b219ec561f19.slice/crio-ef737febd1377ebd136ca52d06def9cbaa9711d3143747eef75f6243f2942767 WatchSource:0}: Error finding container ef737febd1377ebd136ca52d06def9cbaa9711d3143747eef75f6243f2942767: Status 404 returned error can't find the container with id ef737febd1377ebd136ca52d06def9cbaa9711d3143747eef75f6243f2942767
Sep 10 14:45:00.503488 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.503466    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:45:00.525103 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.525075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj" event={"ID":"bf124b00-7259-4e7d-aa67-0b6072adcbd6","Type":"ContainerStarted","Data":"f8efd33821fe539e201294b5a1c33a39dad7741557e3c79accce4b6f8a4d217a"}
Sep 10 14:45:00.526188 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:00.526165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs" event={"ID":"f3f75aad-cd34-4d42-9aad-b219ec561f19","Type":"ContainerStarted","Data":"ef737febd1377ebd136ca52d06def9cbaa9711d3143747eef75f6243f2942767"}
Sep 10 14:45:03.544603 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:03.544553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs" event={"ID":"f3f75aad-cd34-4d42-9aad-b219ec561f19","Type":"ContainerStarted","Data":"783ee920cdbf5e413943309e51de57a40c2050977a49b1bf748d6247d9901f96"}
Sep 10 14:45:03.546098 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:03.546074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj" event={"ID":"bf124b00-7259-4e7d-aa67-0b6072adcbd6","Type":"ContainerStarted","Data":"8b910210e9d16f2e2fe3347e06dc28d1b0366cae7adf3ca7122ab7cb62ce865a"}
Sep 10 14:45:03.559436 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:03.559390    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs" podStartSLOduration=1.217755571 podStartE2EDuration="3.559376631s" totalImagesPullingTime="2.34162106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:03.558566325 +0000 UTC m=+1029.984303746" watchObservedRunningTime="2026-09-10 14:45:03.559376631 +0000 UTC m=+1029.985114050"
Sep 10 14:45:03.572339 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:03.572272    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj" podStartSLOduration=1.201472148 podStartE2EDuration="3.572256063s" totalImagesPullingTime="2.370783915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:03.57138212 +0000 UTC m=+1029.997119539" watchObservedRunningTime="2026-09-10 14:45:03.572256063 +0000 UTC m=+1029.997993479"
Sep 10 14:45:03.735760 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:03.735688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:45:03.735760 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:03.735775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:45:03.748757 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:03.748725    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:45:04.551769 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:04.551733    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf124b00-7259-4e7d-aa67-0b6072adcbd6" containerID="8b910210e9d16f2e2fe3347e06dc28d1b0366cae7adf3ca7122ab7cb62ce865a" exitCode=0
Sep 10 14:45:04.552222 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:04.551820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj" event={"ID":"bf124b00-7259-4e7d-aa67-0b6072adcbd6","Type":"ContainerDied","Data":"8b910210e9d16f2e2fe3347e06dc28d1b0366cae7adf3ca7122ab7cb62ce865a"}
Sep 10 14:45:04.553381 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:04.553361    2566 generic.go:356] "Generic (PLEG): container finished" podID="f3f75aad-cd34-4d42-9aad-b219ec561f19" containerID="783ee920cdbf5e413943309e51de57a40c2050977a49b1bf748d6247d9901f96" exitCode=0
Sep 10 14:45:04.553498 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:04.553425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs" event={"ID":"f3f75aad-cd34-4d42-9aad-b219ec561f19","Type":"ContainerDied","Data":"783ee920cdbf5e413943309e51de57a40c2050977a49b1bf748d6247d9901f96"}
Sep 10 14:45:04.564926 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:04.564905    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:45:05.695437 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:05.695417    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj"
Sep 10 14:45:05.714018 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:05.713995    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"
Sep 10 14:45:06.564203 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:06.564173    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj"
Sep 10 14:45:06.564403 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:06.564220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817525-gmnqj" event={"ID":"bf124b00-7259-4e7d-aa67-0b6072adcbd6","Type":"ContainerDied","Data":"f8efd33821fe539e201294b5a1c33a39dad7741557e3c79accce4b6f8a4d217a"}
Sep 10 14:45:06.564403 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:06.564263    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8efd33821fe539e201294b5a1c33a39dad7741557e3c79accce4b6f8a4d217a"
Sep 10 14:45:06.565532 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:06.565502    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"
Sep 10 14:45:06.565532 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:06.565510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs" event={"ID":"f3f75aad-cd34-4d42-9aad-b219ec561f19","Type":"ContainerDied","Data":"ef737febd1377ebd136ca52d06def9cbaa9711d3143747eef75f6243f2942767"}
Sep 10 14:45:06.565679 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:06.565552    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef737febd1377ebd136ca52d06def9cbaa9711d3143747eef75f6243f2942767"
Sep 10 14:45:11.326890 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.326824    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw" podUID="60fe3397-6d93-40bc-9f89-ba17d88baee3" containerName="main" containerID="cri-o://f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7" gracePeriod=45
Sep 10 14:45:11.418674 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.418638    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"]
Sep 10 14:45:11.419044 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.419030    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f75aad-cd34-4d42-9aad-b219ec561f19" containerName="cleanup"
Sep 10 14:45:11.419093 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.419046    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f75aad-cd34-4d42-9aad-b219ec561f19" containerName="cleanup"
Sep 10 14:45:11.419093 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.419060    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf124b00-7259-4e7d-aa67-0b6072adcbd6" containerName="cleanup"
Sep 10 14:45:11.419093 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.419067    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf124b00-7259-4e7d-aa67-0b6072adcbd6" containerName="cleanup"
Sep 10 14:45:11.419180 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.419131    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f75aad-cd34-4d42-9aad-b219ec561f19" containerName="cleanup"
Sep 10 14:45:11.419180 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.419141    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf124b00-7259-4e7d-aa67-0b6072adcbd6" containerName="cleanup"
Sep 10 14:45:11.422023 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.422007    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.424260 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.424234    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"kube-root-ca.crt\""
Sep 10 14:45:11.424391 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.424283    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"default-dockercfg-cx7lb\""
Sep 10 14:45:11.425036 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.425015    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"openshift-service-ca.crt\""
Sep 10 14:45:11.425247 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.425227    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"distinct-w0-9b1e67-kserve-self-signed-certs\""
Sep 10 14:45:11.431040 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.431016    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"]
Sep 10 14:45:11.556193 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.556161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-model-cache\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.556193 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.556198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-home\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.556381 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.556222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tmp-dir\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.556381 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.556314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-dshm\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.556381 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.556359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tls-certs\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.556507 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.556386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6zm\" (UniqueName: \"kubernetes.io/projected/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-kube-api-access-jg6zm\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.577004 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.576956    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:45:11.591117 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.591086    2566 generic.go:356] "Generic (PLEG): container finished" podID="60fe3397-6d93-40bc-9f89-ba17d88baee3" containerID="f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7" exitCode=0
Sep 10 14:45:11.591244 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.591132    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"
Sep 10 14:45:11.591244 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.591174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw" event={"ID":"60fe3397-6d93-40bc-9f89-ba17d88baee3","Type":"ContainerDied","Data":"f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7"}
Sep 10 14:45:11.591244 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.591227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw" event={"ID":"60fe3397-6d93-40bc-9f89-ba17d88baee3","Type":"ContainerDied","Data":"7be64350a52138f15b29dc03fc095c8e6b3f0a3c3fdbaeb02f4de0696e1019b6"}
Sep 10 14:45:11.591360 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.591251    2566 scope.go:117] "RemoveContainer" containerID="f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7"
Sep 10 14:45:11.602357 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.602337    2566 scope.go:117] "RemoveContainer" containerID="f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7"
Sep 10 14:45:11.602667 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:45:11.602645    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7\": container with ID starting with f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7 not found: ID does not exist" containerID="f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7"
Sep 10 14:45:11.602734 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.602675    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7"} err="failed to get container status \"f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7\": rpc error: code = NotFound desc = could not find container \"f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7\": container with ID starting with f639a37422264a8154ff80189c3b627e4e51f4a13fc2c23cb7a382b44ecc56d7 not found: ID does not exist"
Sep 10 14:45:11.657317 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-dshm\") pod \"60fe3397-6d93-40bc-9f89-ba17d88baee3\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") "
Sep 10 14:45:11.657455 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-home\") pod \"60fe3397-6d93-40bc-9f89-ba17d88baee3\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") "
Sep 10 14:45:11.657455 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657362    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-model-cache\") pod \"60fe3397-6d93-40bc-9f89-ba17d88baee3\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") "
Sep 10 14:45:11.657455 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657396    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-tmp-dir\") pod \"60fe3397-6d93-40bc-9f89-ba17d88baee3\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") "
Sep 10 14:45:11.657455 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4kgd\" (UniqueName: \"kubernetes.io/projected/60fe3397-6d93-40bc-9f89-ba17d88baee3-kube-api-access-p4kgd\") pod \"60fe3397-6d93-40bc-9f89-ba17d88baee3\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") "
Sep 10 14:45:11.657455 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657447    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe3397-6d93-40bc-9f89-ba17d88baee3-tls-certs\") pod \"60fe3397-6d93-40bc-9f89-ba17d88baee3\" (UID: \"60fe3397-6d93-40bc-9f89-ba17d88baee3\") "
Sep 10 14:45:11.657735 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-model-cache\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.657735 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-home\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.657735 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tmp-dir\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.657735 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657617    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-home" (OuterVolumeSpecName: "home") pod "60fe3397-6d93-40bc-9f89-ba17d88baee3" (UID: "60fe3397-6d93-40bc-9f89-ba17d88baee3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:45:11.657735 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657683    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60fe3397-6d93-40bc-9f89-ba17d88baee3" (UID: "60fe3397-6d93-40bc-9f89-ba17d88baee3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:45:11.657971 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-dshm\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.657971 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tls-certs\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.657971 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6zm\" (UniqueName: \"kubernetes.io/projected/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-kube-api-access-jg6zm\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.657971 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.657962    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:11.658162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.658020    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:11.658162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.658103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-model-cache\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.658162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.658130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tmp-dir\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.658288 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.658162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-home\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.658288 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.658264    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-model-cache" (OuterVolumeSpecName: "model-cache") pod "60fe3397-6d93-40bc-9f89-ba17d88baee3" (UID: "60fe3397-6d93-40bc-9f89-ba17d88baee3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:45:11.659814 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.659789    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-dshm" (OuterVolumeSpecName: "dshm") pod "60fe3397-6d93-40bc-9f89-ba17d88baee3" (UID: "60fe3397-6d93-40bc-9f89-ba17d88baee3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:45:11.659814 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.659803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fe3397-6d93-40bc-9f89-ba17d88baee3-kube-api-access-p4kgd" (OuterVolumeSpecName: "kube-api-access-p4kgd") pod "60fe3397-6d93-40bc-9f89-ba17d88baee3" (UID: "60fe3397-6d93-40bc-9f89-ba17d88baee3"). InnerVolumeSpecName "kube-api-access-p4kgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:45:11.659951 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.659881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fe3397-6d93-40bc-9f89-ba17d88baee3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60fe3397-6d93-40bc-9f89-ba17d88baee3" (UID: "60fe3397-6d93-40bc-9f89-ba17d88baee3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:45:11.660507 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.660484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-dshm\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.660759 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.660737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tls-certs\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.665949 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.665927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6zm\" (UniqueName: \"kubernetes.io/projected/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-kube-api-access-jg6zm\") pod \"distinct-w0-9b1e67-kserve-db67b855c-nq7dh\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") " pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.731982 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.731947    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:11.759007 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.758982    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4kgd\" (UniqueName: \"kubernetes.io/projected/60fe3397-6d93-40bc-9f89-ba17d88baee3-kube-api-access-p4kgd\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:11.759007 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.759008    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe3397-6d93-40bc-9f89-ba17d88baee3-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:11.759133 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.759019    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:11.759133 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.759028    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe3397-6d93-40bc-9f89-ba17d88baee3-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:11.915917 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.915884    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"]
Sep 10 14:45:11.919467 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:11.919442    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c072e8/explicit-model-c072e8-kserve-688857695f-5qrzw"]
Sep 10 14:45:12.004872 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:12.002775    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"]
Sep 10 14:45:12.007478 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:45:12.007452    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1c50ec4_3a7b_43dd_afdb_16b000d27b02.slice/crio-9b691d220d8931ba4a04f36873de58550999ff5d39559d360d233f647e928f0e WatchSource:0}: Error finding container 9b691d220d8931ba4a04f36873de58550999ff5d39559d360d233f647e928f0e: Status 404 returned error can't find the container with id 9b691d220d8931ba4a04f36873de58550999ff5d39559d360d233f647e928f0e
Sep 10 14:45:12.111850 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:12.111817    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60fe3397-6d93-40bc-9f89-ba17d88baee3" path="/var/lib/kubelet/pods/60fe3397-6d93-40bc-9f89-ba17d88baee3/volumes"
Sep 10 14:45:12.598143 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:12.598111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh" event={"ID":"c1c50ec4-3a7b-43dd-afdb-16b000d27b02","Type":"ContainerStarted","Data":"353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531"}
Sep 10 14:45:12.598143 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:12.598149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh" event={"ID":"c1c50ec4-3a7b-43dd-afdb-16b000d27b02","Type":"ContainerStarted","Data":"9b691d220d8931ba4a04f36873de58550999ff5d39559d360d233f647e928f0e"}
Sep 10 14:45:12.620589 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:12.620524    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh" podStartSLOduration=1.620510321 podStartE2EDuration="1.620510321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:12.618967277 +0000 UTC m=+1039.044704698" watchObservedRunningTime="2026-09-10 14:45:12.620510321 +0000 UTC m=+1039.046247740"
Sep 10 14:45:21.732640 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:21.732603    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:21.733118 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:21.732670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:21.745632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:21.745606    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:22.660602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:22.660571    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:45:34.377517 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.377438    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"]
Sep 10 14:45:34.378061 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.377734    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz" podUID="54d835be-3716-468f-9f76-58ce2b538d86" containerName="maas-api" containerID="cri-o://0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065" gracePeriod=30
Sep 10 14:45:34.613958 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.613930    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:45:34.712585 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.712556    2566 generic.go:356] "Generic (PLEG): container finished" podID="54d835be-3716-468f-9f76-58ce2b538d86" containerID="0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065" exitCode=0
Sep 10 14:45:34.712790 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.712599    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"
Sep 10 14:45:34.712790 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.712614    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz" event={"ID":"54d835be-3716-468f-9f76-58ce2b538d86","Type":"ContainerDied","Data":"0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065"}
Sep 10 14:45:34.712790 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.712660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz" event={"ID":"54d835be-3716-468f-9f76-58ce2b538d86","Type":"ContainerDied","Data":"c868000674f54950f515a3f29d89a0be2fe7e8b82d024a31daf9237b021f7351"}
Sep 10 14:45:34.712790 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.712682    2566 scope.go:117] "RemoveContainer" containerID="0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065"
Sep 10 14:45:34.721557 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.721539    2566 scope.go:117] "RemoveContainer" containerID="0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065"
Sep 10 14:45:34.721849 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:45:34.721831    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065\": container with ID starting with 0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065 not found: ID does not exist" containerID="0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065"
Sep 10 14:45:34.721912 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.721858    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065"} err="failed to get container status \"0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065\": rpc error: code = NotFound desc = could not find container \"0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065\": container with ID starting with 0bb60e097cb6906275498443de424bb5999291d14a1ac3c97ffd0d18961ac065 not found: ID does not exist"
Sep 10 14:45:34.760313 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.760286    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-metrics-certs\") pod \"54d835be-3716-468f-9f76-58ce2b538d86\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") "
Sep 10 14:45:34.760466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.760385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d835be-3716-468f-9f76-58ce2b538d86-openshift-service-ca\") pod \"54d835be-3716-468f-9f76-58ce2b538d86\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") "
Sep 10 14:45:34.760466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.760415    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g546d\" (UniqueName: \"kubernetes.io/projected/54d835be-3716-468f-9f76-58ce2b538d86-kube-api-access-g546d\") pod \"54d835be-3716-468f-9f76-58ce2b538d86\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") "
Sep 10 14:45:34.760466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.760441    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls\") pod \"54d835be-3716-468f-9f76-58ce2b538d86\" (UID: \"54d835be-3716-468f-9f76-58ce2b538d86\") "
Sep 10 14:45:34.760875 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.760843    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d835be-3716-468f-9f76-58ce2b538d86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54d835be-3716-468f-9f76-58ce2b538d86" (UID: "54d835be-3716-468f-9f76-58ce2b538d86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:45:34.762610 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.762588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54d835be-3716-468f-9f76-58ce2b538d86" (UID: "54d835be-3716-468f-9f76-58ce2b538d86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:45:34.762729 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.762605    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d835be-3716-468f-9f76-58ce2b538d86-kube-api-access-g546d" (OuterVolumeSpecName: "kube-api-access-g546d") pod "54d835be-3716-468f-9f76-58ce2b538d86" (UID: "54d835be-3716-468f-9f76-58ce2b538d86"). InnerVolumeSpecName "kube-api-access-g546d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:45:34.762729 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.762658    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "54d835be-3716-468f-9f76-58ce2b538d86" (UID: "54d835be-3716-468f-9f76-58ce2b538d86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:45:34.861431 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.861399    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d835be-3716-468f-9f76-58ce2b538d86-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:34.861431 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.861428    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g546d\" (UniqueName: \"kubernetes.io/projected/54d835be-3716-468f-9f76-58ce2b538d86-kube-api-access-g546d\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:34.861624 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.861438    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:34.861624 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:34.861450    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d835be-3716-468f-9f76-58ce2b538d86-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:45:35.034151 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:35.034118    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"]
Sep 10 14:45:35.038210 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:35.038176    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-785829b5-6fcd69986c-npdpz"]
Sep 10 14:45:36.110095 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:36.110058    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d835be-3716-468f-9f76-58ce2b538d86" path="/var/lib/kubelet/pods/54d835be-3716-468f-9f76-58ce2b538d86/volumes"
Sep 10 14:45:58.542178 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.542142    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"]
Sep 10 14:45:58.542595 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.542501    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60fe3397-6d93-40bc-9f89-ba17d88baee3" containerName="main"
Sep 10 14:45:58.542595 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.542512    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="60fe3397-6d93-40bc-9f89-ba17d88baee3" containerName="main"
Sep 10 14:45:58.542595 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.542520    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d835be-3716-468f-9f76-58ce2b538d86" containerName="maas-api"
Sep 10 14:45:58.542595 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.542526    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d835be-3716-468f-9f76-58ce2b538d86" containerName="maas-api"
Sep 10 14:45:58.542595 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.542574    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d835be-3716-468f-9f76-58ce2b538d86" containerName="maas-api"
Sep 10 14:45:58.542595 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.542583    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="60fe3397-6d93-40bc-9f89-ba17d88baee3" containerName="main"
Sep 10 14:45:58.547046 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.547023    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.549034 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.549008    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5c9641-serving-cert\""
Sep 10 14:45:58.558599 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.558572    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"]
Sep 10 14:45:58.667071 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.667032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.667071 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.667077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.667291 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.667167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kkb\" (UniqueName: \"kubernetes.io/projected/e3df5b42-c1ef-453f-91b2-4813591b49ef-kube-api-access-d4kkb\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.667291 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.667235    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3df5b42-c1ef-453f-91b2-4813591b49ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.768315 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.768274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.768499 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.768346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kkb\" (UniqueName: \"kubernetes.io/projected/e3df5b42-c1ef-453f-91b2-4813591b49ef-kube-api-access-d4kkb\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.768499 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.768379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3df5b42-c1ef-453f-91b2-4813591b49ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.768499 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.768426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.769097 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.769069    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3df5b42-c1ef-453f-91b2-4813591b49ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.770826 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.770806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.770876 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.770853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.779592 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.779563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kkb\" (UniqueName: \"kubernetes.io/projected/e3df5b42-c1ef-453f-91b2-4813591b49ef-kube-api-access-d4kkb\") pod \"maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:58.858104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:58.858003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:45:59.130337 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:59.130297    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"]
Sep 10 14:45:59.132823 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:45:59.132792    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3df5b42_c1ef_453f_91b2_4813591b49ef.slice/crio-8a08b160fbf4fd0b0d1da9a05e184c9f66e2551e154080172f86ce494dcc70b1 WatchSource:0}: Error finding container 8a08b160fbf4fd0b0d1da9a05e184c9f66e2551e154080172f86ce494dcc70b1: Status 404 returned error can't find the container with id 8a08b160fbf4fd0b0d1da9a05e184c9f66e2551e154080172f86ce494dcc70b1
Sep 10 14:45:59.134107 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:59.134091    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:45:59.833575 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:45:59.833551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p" event={"ID":"e3df5b42-c1ef-453f-91b2-4813591b49ef","Type":"ContainerStarted","Data":"8a08b160fbf4fd0b0d1da9a05e184c9f66e2551e154080172f86ce494dcc70b1"}
Sep 10 14:46:00.839357 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:00.839321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p" event={"ID":"e3df5b42-c1ef-453f-91b2-4813591b49ef","Type":"ContainerStarted","Data":"0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454"}
Sep 10 14:46:00.839750 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:00.839461    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:46:00.856275 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:00.856221    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p" podStartSLOduration=2.159379774 podStartE2EDuration="2.856207662s" totalImagesPullingTime="696.827888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:00.8552605 +0000 UTC m=+1087.280997919" watchObservedRunningTime="2026-09-10 14:46:00.856207662 +0000 UTC m=+1087.281945081"
Sep 10 14:46:06.849134 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:06.849101    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:46:23.242162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.242125    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"]
Sep 10 14:46:23.242604 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.242322    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6" podUID="baaa519d-3dac-4f6d-86ba-0ea3e0d84636" containerName="payload-processing" containerID="cri-o://f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66" gracePeriod=30
Sep 10 14:46:23.495884 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.495830    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:46:23.589632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.589598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9tfx\" (UniqueName: \"kubernetes.io/projected/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-kube-api-access-q9tfx\") pod \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\" (UID: \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\") "
Sep 10 14:46:23.589632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.589642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-plugins-config-volume\") pod \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\" (UID: \"baaa519d-3dac-4f6d-86ba-0ea3e0d84636\") "
Sep 10 14:46:23.590078 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.590047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "baaa519d-3dac-4f6d-86ba-0ea3e0d84636" (UID: "baaa519d-3dac-4f6d-86ba-0ea3e0d84636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:46:23.591912 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.591879    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-kube-api-access-q9tfx" (OuterVolumeSpecName: "kube-api-access-q9tfx") pod "baaa519d-3dac-4f6d-86ba-0ea3e0d84636" (UID: "baaa519d-3dac-4f6d-86ba-0ea3e0d84636"). InnerVolumeSpecName "kube-api-access-q9tfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:46:23.690602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.690565    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9tfx\" (UniqueName: \"kubernetes.io/projected/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-kube-api-access-q9tfx\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:23.690602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.690598    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa519d-3dac-4f6d-86ba-0ea3e0d84636-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:23.945787 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.945751    2566 generic.go:356] "Generic (PLEG): container finished" podID="baaa519d-3dac-4f6d-86ba-0ea3e0d84636" containerID="f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66" exitCode=0
Sep 10 14:46:23.945962 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.945798    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"
Sep 10 14:46:23.945962 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.945832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6" event={"ID":"baaa519d-3dac-4f6d-86ba-0ea3e0d84636","Type":"ContainerDied","Data":"f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66"}
Sep 10 14:46:23.945962 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.945885    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6" event={"ID":"baaa519d-3dac-4f6d-86ba-0ea3e0d84636","Type":"ContainerDied","Data":"6ddf7bc0349946ef19fed20a71a7fdaf2889caa9a25f8c897d08b4a47c224875"}
Sep 10 14:46:23.945962 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.945909    2566 scope.go:117] "RemoveContainer" containerID="f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66"
Sep 10 14:46:23.955110 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.954999    2566 scope.go:117] "RemoveContainer" containerID="f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66"
Sep 10 14:46:23.955367 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:23.955345    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66\": container with ID starting with f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66 not found: ID does not exist" containerID="f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66"
Sep 10 14:46:23.955483 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.955374    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66"} err="failed to get container status \"f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66\": rpc error: code = NotFound desc = could not find container \"f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66\": container with ID starting with f7266835a310c6e789e49d17b2ac6be066b82d060ef12d4be46b70ec5cfdfb66 not found: ID does not exist"
Sep 10 14:46:23.967767 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.967746    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"]
Sep 10 14:46:23.971801 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:23.971779    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5121f4-7c946b66c9-rl6x6"]
Sep 10 14:46:24.112428 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:24.112398    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baaa519d-3dac-4f6d-86ba-0ea3e0d84636" path="/var/lib/kubelet/pods/baaa519d-3dac-4f6d-86ba-0ea3e0d84636/volumes"
Sep 10 14:46:28.766718 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.766665    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"]
Sep 10 14:46:28.767255 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.767170    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baaa519d-3dac-4f6d-86ba-0ea3e0d84636" containerName="payload-processing"
Sep 10 14:46:28.767255 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.767191    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="baaa519d-3dac-4f6d-86ba-0ea3e0d84636" containerName="payload-processing"
Sep 10 14:46:28.767388 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.767297    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="baaa519d-3dac-4f6d-86ba-0ea3e0d84636" containerName="payload-processing"
Sep 10 14:46:28.771615 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.771594    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:28.773926 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.773900    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ecfa25\""
Sep 10 14:46:28.781041 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.781020    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"]
Sep 10 14:46:28.934602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.934554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rc5\" (UniqueName: \"kubernetes.io/projected/4ca354d8-9d2d-4c48-a731-a44be67b3d39-kube-api-access-d4rc5\") pod \"payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9\" (UID: \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:28.934978 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:28.934938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca354d8-9d2d-4c48-a731-a44be67b3d39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9\" (UID: \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:29.035492 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.035396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca354d8-9d2d-4c48-a731-a44be67b3d39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9\" (UID: \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:29.035492 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.035484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rc5\" (UniqueName: \"kubernetes.io/projected/4ca354d8-9d2d-4c48-a731-a44be67b3d39-kube-api-access-d4rc5\") pod \"payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9\" (UID: \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:29.036184 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.036156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca354d8-9d2d-4c48-a731-a44be67b3d39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9\" (UID: \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:29.043117 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.043095    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rc5\" (UniqueName: \"kubernetes.io/projected/4ca354d8-9d2d-4c48-a731-a44be67b3d39-kube-api-access-d4rc5\") pod \"payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9\" (UID: \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:29.083098 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.083070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:29.366783 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.366745    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"]
Sep 10 14:46:29.368032 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:46:29.368007    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca354d8_9d2d_4c48_a731_a44be67b3d39.slice/crio-b8dc5db10af902d4d55ffd4c8270c3ca63e3a99c21d71c09dcd25f09cc895f25 WatchSource:0}: Error finding container b8dc5db10af902d4d55ffd4c8270c3ca63e3a99c21d71c09dcd25f09cc895f25: Status 404 returned error can't find the container with id b8dc5db10af902d4d55ffd4c8270c3ca63e3a99c21d71c09dcd25f09cc895f25
Sep 10 14:46:29.976361 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.976326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9" event={"ID":"4ca354d8-9d2d-4c48-a731-a44be67b3d39","Type":"ContainerStarted","Data":"65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa"}
Sep 10 14:46:29.976795 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.976378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9" event={"ID":"4ca354d8-9d2d-4c48-a731-a44be67b3d39","Type":"ContainerStarted","Data":"b8dc5db10af902d4d55ffd4c8270c3ca63e3a99c21d71c09dcd25f09cc895f25"}
Sep 10 14:46:29.976795 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:29.976539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:30.001960 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:30.001907    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9" podStartSLOduration=2.001887089 podStartE2EDuration="2.001887089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:29.998939255 +0000 UTC m=+1116.424676676" watchObservedRunningTime="2026-09-10 14:46:30.001887089 +0000 UTC m=+1116.427624510"
Sep 10 14:46:40.983662 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:40.983628    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:46:41.209379 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.209344    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"]
Sep 10 14:46:41.209626 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.209601    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp" podUID="3842443f-d383-4466-8d68-4a4755e07e67" containerName="maas-api" containerID="cri-o://8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707" gracePeriod=30
Sep 10 14:46:41.236790 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.236724    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"]
Sep 10 14:46:41.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.238178    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2dccd5-29817525-tdsfs"]
Sep 10 14:46:41.259594 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.259570    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"]
Sep 10 14:46:41.259806 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.259782    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m" podUID="bc8c3982-245a-4d7f-a23c-64f8ba95d39d" containerName="payload-processing" containerID="cri-o://9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64" gracePeriod=30
Sep 10 14:46:41.266065 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.266037    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"]
Sep 10 14:46:41.266310 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.266252    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk" podUID="6dbd3ba9-6106-47b5-91f6-20b40e3946e6" containerName="payload-pre-processing" containerID="cri-o://e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce" gracePeriod=30
Sep 10 14:46:41.521664 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.521640    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:46:41.524540 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.524520    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:46:41.533113 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.533090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-metrics-certs\") pod \"3842443f-d383-4466-8d68-4a4755e07e67\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") "
Sep 10 14:46:41.533232 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.533138    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-plugins-config-volume\") pod \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\" (UID: \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\") "
Sep 10 14:46:41.533232 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.533174    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3842443f-d383-4466-8d68-4a4755e07e67-openshift-service-ca\") pod \"3842443f-d383-4466-8d68-4a4755e07e67\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") "
Sep 10 14:46:41.533232 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.533214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvq27\" (UniqueName: \"kubernetes.io/projected/3842443f-d383-4466-8d68-4a4755e07e67-kube-api-access-fvq27\") pod \"3842443f-d383-4466-8d68-4a4755e07e67\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") "
Sep 10 14:46:41.533388 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.533307    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-maas-api-tls\") pod \"3842443f-d383-4466-8d68-4a4755e07e67\" (UID: \"3842443f-d383-4466-8d68-4a4755e07e67\") "
Sep 10 14:46:41.533388 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.533344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jhlx\" (UniqueName: \"kubernetes.io/projected/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-kube-api-access-9jhlx\") pod \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\" (UID: \"bc8c3982-245a-4d7f-a23c-64f8ba95d39d\") "
Sep 10 14:46:41.533577 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.533550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3842443f-d383-4466-8d68-4a4755e07e67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3842443f-d383-4466-8d68-4a4755e07e67" (UID: "3842443f-d383-4466-8d68-4a4755e07e67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:46:41.533577 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.533569    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc8c3982-245a-4d7f-a23c-64f8ba95d39d" (UID: "bc8c3982-245a-4d7f-a23c-64f8ba95d39d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:46:41.535241 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.535216    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3842443f-d383-4466-8d68-4a4755e07e67" (UID: "3842443f-d383-4466-8d68-4a4755e07e67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:46:41.535387 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.535371    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3842443f-d383-4466-8d68-4a4755e07e67-kube-api-access-fvq27" (OuterVolumeSpecName: "kube-api-access-fvq27") pod "3842443f-d383-4466-8d68-4a4755e07e67" (UID: "3842443f-d383-4466-8d68-4a4755e07e67"). InnerVolumeSpecName "kube-api-access-fvq27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:46:41.535432 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.535404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-kube-api-access-9jhlx" (OuterVolumeSpecName: "kube-api-access-9jhlx") pod "bc8c3982-245a-4d7f-a23c-64f8ba95d39d" (UID: "bc8c3982-245a-4d7f-a23c-64f8ba95d39d"). InnerVolumeSpecName "kube-api-access-9jhlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:46:41.535600 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.535580    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3842443f-d383-4466-8d68-4a4755e07e67" (UID: "3842443f-d383-4466-8d68-4a4755e07e67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:46:41.554042 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.554000    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:46:41.634702 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.634668    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp2jt\" (UniqueName: \"kubernetes.io/projected/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-kube-api-access-kp2jt\") pod \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\" (UID: \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\") "
Sep 10 14:46:41.634832 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.634739    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-plugins-config-volume\") pod \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\" (UID: \"6dbd3ba9-6106-47b5-91f6-20b40e3946e6\") "
Sep 10 14:46:41.634937 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.634925    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:41.634973 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.634940    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jhlx\" (UniqueName: \"kubernetes.io/projected/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-kube-api-access-9jhlx\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:41.634973 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.634949    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3842443f-d383-4466-8d68-4a4755e07e67-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:41.634973 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.634957    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c3982-245a-4d7f-a23c-64f8ba95d39d-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:41.634973 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.634966    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3842443f-d383-4466-8d68-4a4755e07e67-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:41.635129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.634974    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvq27\" (UniqueName: \"kubernetes.io/projected/3842443f-d383-4466-8d68-4a4755e07e67-kube-api-access-fvq27\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:41.635129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.635068    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dbd3ba9-6106-47b5-91f6-20b40e3946e6" (UID: "6dbd3ba9-6106-47b5-91f6-20b40e3946e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:46:41.636625 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.636604    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-kube-api-access-kp2jt" (OuterVolumeSpecName: "kube-api-access-kp2jt") pod "6dbd3ba9-6106-47b5-91f6-20b40e3946e6" (UID: "6dbd3ba9-6106-47b5-91f6-20b40e3946e6"). InnerVolumeSpecName "kube-api-access-kp2jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:46:41.735889 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.735859    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2jt\" (UniqueName: \"kubernetes.io/projected/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-kube-api-access-kp2jt\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:41.735889 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:41.735883    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd3ba9-6106-47b5-91f6-20b40e3946e6-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:42.038126 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.038095    2566 generic.go:356] "Generic (PLEG): container finished" podID="3842443f-d383-4466-8d68-4a4755e07e67" containerID="8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707" exitCode=0
Sep 10 14:46:42.038542 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.038140    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"
Sep 10 14:46:42.038542 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.038181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp" event={"ID":"3842443f-d383-4466-8d68-4a4755e07e67","Type":"ContainerDied","Data":"8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707"}
Sep 10 14:46:42.038542 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.038232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp" event={"ID":"3842443f-d383-4466-8d68-4a4755e07e67","Type":"ContainerDied","Data":"f189380527e4b79e3c98f0bb6c7d59e69b0f52313b45c2b5a03d5e3729b5f695"}
Sep 10 14:46:42.038542 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.038264    2566 scope.go:117] "RemoveContainer" containerID="8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707"
Sep 10 14:46:42.039615 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.039594    2566 generic.go:356] "Generic (PLEG): container finished" podID="bc8c3982-245a-4d7f-a23c-64f8ba95d39d" containerID="9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64" exitCode=0
Sep 10 14:46:42.039730 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.039633    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"
Sep 10 14:46:42.039730 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.039658    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m" event={"ID":"bc8c3982-245a-4d7f-a23c-64f8ba95d39d","Type":"ContainerDied","Data":"9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64"}
Sep 10 14:46:42.039730 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.039717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m" event={"ID":"bc8c3982-245a-4d7f-a23c-64f8ba95d39d","Type":"ContainerDied","Data":"6abf670d65b4df208d98883ad81e373034df89cf96929f9a34dd582ccd21ee10"}
Sep 10 14:46:42.041190 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.041164    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dbd3ba9-6106-47b5-91f6-20b40e3946e6" containerID="e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce" exitCode=0
Sep 10 14:46:42.041323 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.041201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk" event={"ID":"6dbd3ba9-6106-47b5-91f6-20b40e3946e6","Type":"ContainerDied","Data":"e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce"}
Sep 10 14:46:42.041323 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.041236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk" event={"ID":"6dbd3ba9-6106-47b5-91f6-20b40e3946e6","Type":"ContainerDied","Data":"227e93e804d0629514d290b0ec1298baabf960967512ce106d37dde2eccc0007"}
Sep 10 14:46:42.041323 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.041290    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"
Sep 10 14:46:42.048942 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.048920    2566 scope.go:117] "RemoveContainer" containerID="8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707"
Sep 10 14:46:42.049295 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:42.049267    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707\": container with ID starting with 8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707 not found: ID does not exist" containerID="8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707"
Sep 10 14:46:42.049369 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.049306    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707"} err="failed to get container status \"8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707\": rpc error: code = NotFound desc = could not find container \"8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707\": container with ID starting with 8c13dc757dd737176c3c89fc8a7aad2fc177cbf751770cd663c040fd6f422707 not found: ID does not exist"
Sep 10 14:46:42.049369 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.049330    2566 scope.go:117] "RemoveContainer" containerID="9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64"
Sep 10 14:46:42.057505 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.057488    2566 scope.go:117] "RemoveContainer" containerID="9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64"
Sep 10 14:46:42.057787 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:42.057767    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64\": container with ID starting with 9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64 not found: ID does not exist" containerID="9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64"
Sep 10 14:46:42.057873 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.057791    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64"} err="failed to get container status \"9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64\": rpc error: code = NotFound desc = could not find container \"9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64\": container with ID starting with 9b543d124511db1c33e95fc78fff9bd36a38736ab18a15544dbbbc3148e31b64 not found: ID does not exist"
Sep 10 14:46:42.057873 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.057806    2566 scope.go:117] "RemoveContainer" containerID="e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce"
Sep 10 14:46:42.062033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.062014    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"]
Sep 10 14:46:42.065505 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.065485    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp"]
Sep 10 14:46:42.065779 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.065751    2566 scope.go:117] "RemoveContainer" containerID="e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce"
Sep 10 14:46:42.066040 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:42.066025    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce\": container with ID starting with e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce not found: ID does not exist" containerID="e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce"
Sep 10 14:46:42.066101 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.066045    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce"} err="failed to get container status \"e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce\": rpc error: code = NotFound desc = could not find container \"e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce\": container with ID starting with e886f7f129e9e7238a5b21396f212d1a5db52a6c41e61eb83ec0e678510561ce not found: ID does not exist"
Sep 10 14:46:42.074277 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.074255    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"]
Sep 10 14:46:42.076825 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.076806    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2dccd5-5bdc795bb6-mj6wk"]
Sep 10 14:46:42.085457 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.085437    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"]
Sep 10 14:46:42.090955 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.090934    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2dccd5-5797cc8fcc-ksb7m"]
Sep 10 14:46:42.110496 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.110472    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3842443f-d383-4466-8d68-4a4755e07e67" path="/var/lib/kubelet/pods/3842443f-d383-4466-8d68-4a4755e07e67/volumes"
Sep 10 14:46:42.110874 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.110860    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dbd3ba9-6106-47b5-91f6-20b40e3946e6" path="/var/lib/kubelet/pods/6dbd3ba9-6106-47b5-91f6-20b40e3946e6/volumes"
Sep 10 14:46:42.111191 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.111179    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc8c3982-245a-4d7f-a23c-64f8ba95d39d" path="/var/lib/kubelet/pods/bc8c3982-245a-4d7f-a23c-64f8ba95d39d/volumes"
Sep 10 14:46:42.111432 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:42.111420    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f75aad-cd34-4d42-9aad-b219ec561f19" path="/var/lib/kubelet/pods/f3f75aad-cd34-4d42-9aad-b219ec561f19/volumes"
Sep 10 14:46:44.435583 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:44.435537    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2dccd5-7cdd4bcb7f-f2zbp" podUID="3842443f-d383-4466-8d68-4a4755e07e67" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.29:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:46:52.471062 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471025    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7818-0cd35a"]
Sep 10 14:46:52.471518 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471497    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8c3982-245a-4d7f-a23c-64f8ba95d39d" containerName="payload-processing"
Sep 10 14:46:52.471518 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471517    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8c3982-245a-4d7f-a23c-64f8ba95d39d" containerName="payload-processing"
Sep 10 14:46:52.471627 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471532    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3842443f-d383-4466-8d68-4a4755e07e67" containerName="maas-api"
Sep 10 14:46:52.471627 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471540    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3842443f-d383-4466-8d68-4a4755e07e67" containerName="maas-api"
Sep 10 14:46:52.471627 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471561    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dbd3ba9-6106-47b5-91f6-20b40e3946e6" containerName="payload-pre-processing"
Sep 10 14:46:52.471627 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471572    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dbd3ba9-6106-47b5-91f6-20b40e3946e6" containerName="payload-pre-processing"
Sep 10 14:46:52.471844 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471665    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3842443f-d383-4466-8d68-4a4755e07e67" containerName="maas-api"
Sep 10 14:46:52.471844 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471679    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8c3982-245a-4d7f-a23c-64f8ba95d39d" containerName="payload-processing"
Sep 10 14:46:52.471844 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.471706    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dbd3ba9-6106-47b5-91f6-20b40e3946e6" containerName="payload-pre-processing"
Sep 10 14:46:52.474630 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.474609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-0cd35a"
Sep 10 14:46:52.476739 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.476718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x6mfq\""
Sep 10 14:46:52.477916 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.477890    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7818-0cd35a"]
Sep 10 14:46:52.524536 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.524507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcn2\" (UniqueName: \"kubernetes.io/projected/fb46f3f8-c7cf-4844-ae15-d731ddef5dd4-kube-api-access-9bcn2\") pod \"test-curl-7818-0cd35a\" (UID: \"fb46f3f8-c7cf-4844-ae15-d731ddef5dd4\") " pod="opendatahub/test-curl-7818-0cd35a"
Sep 10 14:46:52.625797 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.625753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcn2\" (UniqueName: \"kubernetes.io/projected/fb46f3f8-c7cf-4844-ae15-d731ddef5dd4-kube-api-access-9bcn2\") pod \"test-curl-7818-0cd35a\" (UID: \"fb46f3f8-c7cf-4844-ae15-d731ddef5dd4\") " pod="opendatahub/test-curl-7818-0cd35a"
Sep 10 14:46:52.633917 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.633893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcn2\" (UniqueName: \"kubernetes.io/projected/fb46f3f8-c7cf-4844-ae15-d731ddef5dd4-kube-api-access-9bcn2\") pod \"test-curl-7818-0cd35a\" (UID: \"fb46f3f8-c7cf-4844-ae15-d731ddef5dd4\") " pod="opendatahub/test-curl-7818-0cd35a"
Sep 10 14:46:52.784562 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:52.784480    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-0cd35a"
Sep 10 14:46:53.050124 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:53.050095    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7818-0cd35a"]
Sep 10 14:46:53.052491 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:46:53.052462    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb46f3f8_c7cf_4844_ae15_d731ddef5dd4.slice/crio-8d688b80c979dbd142ba56152de369a94e4ecadb01f5ee4c83d0c5ecd988edb6 WatchSource:0}: Error finding container 8d688b80c979dbd142ba56152de369a94e4ecadb01f5ee4c83d0c5ecd988edb6: Status 404 returned error can't find the container with id 8d688b80c979dbd142ba56152de369a94e4ecadb01f5ee4c83d0c5ecd988edb6
Sep 10 14:46:53.093663 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:53.093636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7818-0cd35a" event={"ID":"fb46f3f8-c7cf-4844-ae15-d731ddef5dd4","Type":"ContainerStarted","Data":"8d688b80c979dbd142ba56152de369a94e4ecadb01f5ee4c83d0c5ecd988edb6"}
Sep 10 14:46:54.099019 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.098988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7818-0cd35a" event={"ID":"fb46f3f8-c7cf-4844-ae15-d731ddef5dd4","Type":"ContainerStarted","Data":"c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b"}
Sep 10 14:46:54.116841 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.116580    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7818-0cd35a" podStartSLOduration=2.116559509 podStartE2EDuration="2.116559509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:54.111567577 +0000 UTC m=+1140.537304997" watchObservedRunningTime="2026-09-10 14:46:54.116559509 +0000 UTC m=+1140.542296929"
Sep 10 14:46:54.699334 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.699302    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7818-0cd35a"]
Sep 10 14:46:54.704392 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.704362    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7818-0cd35a"]
Sep 10 14:46:54.823513 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.823443    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7818-05a88a"]
Sep 10 14:46:54.823804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.823789    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb46f3f8-c7cf-4844-ae15-d731ddef5dd4" containerName="test-curl-7818-0cd35a"
Sep 10 14:46:54.823859 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.823804    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb46f3f8-c7cf-4844-ae15-d731ddef5dd4" containerName="test-curl-7818-0cd35a"
Sep 10 14:46:54.823897 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.823859    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb46f3f8-c7cf-4844-ae15-d731ddef5dd4" containerName="test-curl-7818-0cd35a"
Sep 10 14:46:54.828399 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.828382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-05a88a"
Sep 10 14:46:54.832246 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.832223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7818-05a88a"]
Sep 10 14:46:54.945773 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:54.945736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbvj\" (UniqueName: \"kubernetes.io/projected/c190b1a1-4a24-4b21-baec-97aca8181db7-kube-api-access-qkbvj\") pod \"test-curl-7818-05a88a\" (UID: \"c190b1a1-4a24-4b21-baec-97aca8181db7\") " pod="opendatahub/test-curl-7818-05a88a"
Sep 10 14:46:55.046480 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:55.046445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbvj\" (UniqueName: \"kubernetes.io/projected/c190b1a1-4a24-4b21-baec-97aca8181db7-kube-api-access-qkbvj\") pod \"test-curl-7818-05a88a\" (UID: \"c190b1a1-4a24-4b21-baec-97aca8181db7\") " pod="opendatahub/test-curl-7818-05a88a"
Sep 10 14:46:55.053725 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:55.053677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbvj\" (UniqueName: \"kubernetes.io/projected/c190b1a1-4a24-4b21-baec-97aca8181db7-kube-api-access-qkbvj\") pod \"test-curl-7818-05a88a\" (UID: \"c190b1a1-4a24-4b21-baec-97aca8181db7\") " pod="opendatahub/test-curl-7818-05a88a"
Sep 10 14:46:55.103675 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:55.103614    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7818-0cd35a" container="test-curl-7818-0cd35a"
Sep 10 14:46:55.138170 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:55.138148    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-05a88a"
Sep 10 14:46:55.402501 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:55.402476    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7818-05a88a"]
Sep 10 14:46:55.404560 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:46:55.404532    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc190b1a1_4a24_4b21_baec_97aca8181db7.slice/crio-9cbb5759a38a9bb4c016e73e6071ecf3284b4f7f1e516c6989ce194784a30fcb WatchSource:0}: Error finding container 9cbb5759a38a9bb4c016e73e6071ecf3284b4f7f1e516c6989ce194784a30fcb: Status 404 returned error can't find the container with id 9cbb5759a38a9bb4c016e73e6071ecf3284b4f7f1e516c6989ce194784a30fcb
Sep 10 14:46:56.110586 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.110542    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7818-0cd35a" podUID="fb46f3f8-c7cf-4844-ae15-d731ddef5dd4" containerName="test-curl-7818-0cd35a" containerID="cri-o://c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b" gracePeriod=2
Sep 10 14:46:56.111039 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.110606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7818-05a88a" event={"ID":"c190b1a1-4a24-4b21-baec-97aca8181db7","Type":"ContainerStarted","Data":"b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee"}
Sep 10 14:46:56.111039 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.110637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7818-05a88a" event={"ID":"c190b1a1-4a24-4b21-baec-97aca8181db7","Type":"ContainerStarted","Data":"9cbb5759a38a9bb4c016e73e6071ecf3284b4f7f1e516c6989ce194784a30fcb"}
Sep 10 14:46:56.124524 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.124481    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7818-05a88a" podStartSLOduration=2.124461873 podStartE2EDuration="2.124461873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:56.124147019 +0000 UTC m=+1142.549884440" watchObservedRunningTime="2026-09-10 14:46:56.124461873 +0000 UTC m=+1142.550199293"
Sep 10 14:46:56.721502 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.721469    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7818-05a88a"]
Sep 10 14:46:56.725512 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.725490    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7818-05a88a"]
Sep 10 14:46:56.848186 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.848118    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7818-8edad1"]
Sep 10 14:46:56.848436 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.848424    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c190b1a1-4a24-4b21-baec-97aca8181db7" containerName="test-curl-7818-05a88a"
Sep 10 14:46:56.848484 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.848437    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c190b1a1-4a24-4b21-baec-97aca8181db7" containerName="test-curl-7818-05a88a"
Sep 10 14:46:56.848519 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.848486    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c190b1a1-4a24-4b21-baec-97aca8181db7" containerName="test-curl-7818-05a88a"
Sep 10 14:46:56.850956 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.850941    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:46:56.854180 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.854147    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7818-8edad1"]
Sep 10 14:46:56.924899 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.924864    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"]
Sep 10 14:46:56.928658 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.928628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:56.934543 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.934425    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b20fba\"/\"kube-root-ca.crt\""
Sep 10 14:46:56.934543 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.934451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b20fba\"/\"noten-model-b20fba-kserve-self-signed-certs\""
Sep 10 14:46:56.934809 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.934783    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b20fba\"/\"default-dockercfg-zmfl6\""
Sep 10 14:46:56.934942 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.934895    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b20fba\"/\"openshift-service-ca.crt\""
Sep 10 14:46:56.939471 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.939445    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"]
Sep 10 14:46:56.962757 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.962726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-home\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:56.962874 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.962765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-dshm\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:56.962874 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.962790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8fc\" (UniqueName: \"kubernetes.io/projected/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-kube-api-access-wf8fc\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:56.962874 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.962853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-model-cache\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:56.962977 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.962889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzmp\" (UniqueName: \"kubernetes.io/projected/e2e221f3-6f1a-466a-b671-7ceb103d1237-kube-api-access-lxzmp\") pod \"test-curl-7818-8edad1\" (UID: \"e2e221f3-6f1a-466a-b671-7ceb103d1237\") " pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:46:56.962977 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.962921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tls-certs\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:56.962977 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:56.962973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tmp-dir\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063372 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-model-cache\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzmp\" (UniqueName: \"kubernetes.io/projected/e2e221f3-6f1a-466a-b671-7ceb103d1237-kube-api-access-lxzmp\") pod \"test-curl-7818-8edad1\" (UID: \"e2e221f3-6f1a-466a-b671-7ceb103d1237\") " pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:46:57.063539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tls-certs\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tmp-dir\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-home\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063537    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-dshm\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8fc\" (UniqueName: \"kubernetes.io/projected/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-kube-api-access-wf8fc\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-model-cache\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063919 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tmp-dir\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.063966 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.063952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-home\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.065740 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.065721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-dshm\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.065913 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.065896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tls-certs\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.070796 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.070773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzmp\" (UniqueName: \"kubernetes.io/projected/e2e221f3-6f1a-466a-b671-7ceb103d1237-kube-api-access-lxzmp\") pod \"test-curl-7818-8edad1\" (UID: \"e2e221f3-6f1a-466a-b671-7ceb103d1237\") " pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:46:57.070890 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.070845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8fc\" (UniqueName: \"kubernetes.io/projected/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-kube-api-access-wf8fc\") pod \"noten-model-b20fba-kserve-5c884cf555-5lhtk\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") " pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.115063 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:57.115003    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7818-05a88a" container="test-curl-7818-05a88a"
Sep 10 14:46:57.160418 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.160380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:46:57.245517 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.245483    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:46:57.430394 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.430371    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7818-8edad1"]
Sep 10 14:46:57.432157 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:46:57.432129    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2e221f3_6f1a_466a_b671_7ceb103d1237.slice/crio-9a88e5ace1fcb82e0d5886003d1823a5b88d1b6d206b691f82ee81c1e53b6020 WatchSource:0}: Error finding container 9a88e5ace1fcb82e0d5886003d1823a5b88d1b6d206b691f82ee81c1e53b6020: Status 404 returned error can't find the container with id 9a88e5ace1fcb82e0d5886003d1823a5b88d1b6d206b691f82ee81c1e53b6020
Sep 10 14:46:57.520557 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:57.520537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"]
Sep 10 14:46:57.522871 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:46:57.522839    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode378fa4d_e0f9_4d3c_9d06_65ada3704e5e.slice/crio-07635ea1de56fa095577f8440322ff8b813e8593a2e4882a85d6588e676e8d3f WatchSource:0}: Error finding container 07635ea1de56fa095577f8440322ff8b813e8593a2e4882a85d6588e676e8d3f: Status 404 returned error can't find the container with id 07635ea1de56fa095577f8440322ff8b813e8593a2e4882a85d6588e676e8d3f
Sep 10 14:46:58.121170 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.121143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7818-8edad1" event={"ID":"e2e221f3-6f1a-466a-b671-7ceb103d1237","Type":"ContainerStarted","Data":"ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5"}
Sep 10 14:46:58.121472 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.121185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7818-8edad1" event={"ID":"e2e221f3-6f1a-466a-b671-7ceb103d1237","Type":"ContainerStarted","Data":"9a88e5ace1fcb82e0d5886003d1823a5b88d1b6d206b691f82ee81c1e53b6020"}
Sep 10 14:46:58.122886 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.122820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk" event={"ID":"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e","Type":"ContainerStarted","Data":"e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746"}
Sep 10 14:46:58.122886 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.122853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk" event={"ID":"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e","Type":"ContainerStarted","Data":"07635ea1de56fa095577f8440322ff8b813e8593a2e4882a85d6588e676e8d3f"}
Sep 10 14:46:58.122886 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.122864    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7818-05a88a" podUID="c190b1a1-4a24-4b21-baec-97aca8181db7" containerName="test-curl-7818-05a88a" containerID="cri-o://b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee" gracePeriod=2
Sep 10 14:46:58.135295 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.135251    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7818-8edad1" podStartSLOduration=2.135237916 podStartE2EDuration="2.135237916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:58.133879793 +0000 UTC m=+1144.559617213" watchObservedRunningTime="2026-09-10 14:46:58.135237916 +0000 UTC m=+1144.560975335"
Sep 10 14:46:58.149734 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.149672    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk" podStartSLOduration=2.149660445 podStartE2EDuration="2.149660445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:58.148289446 +0000 UTC m=+1144.574026865" watchObservedRunningTime="2026-09-10 14:46:58.149660445 +0000 UTC m=+1144.575397890"
Sep 10 14:46:58.298499 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.298475    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-0cd35a"
Sep 10 14:46:58.372365 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.372303    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bcn2\" (UniqueName: \"kubernetes.io/projected/fb46f3f8-c7cf-4844-ae15-d731ddef5dd4-kube-api-access-9bcn2\") pod \"fb46f3f8-c7cf-4844-ae15-d731ddef5dd4\" (UID: \"fb46f3f8-c7cf-4844-ae15-d731ddef5dd4\") "
Sep 10 14:46:58.374094 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.374072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb46f3f8-c7cf-4844-ae15-d731ddef5dd4-kube-api-access-9bcn2" (OuterVolumeSpecName: "kube-api-access-9bcn2") pod "fb46f3f8-c7cf-4844-ae15-d731ddef5dd4" (UID: "fb46f3f8-c7cf-4844-ae15-d731ddef5dd4"). InnerVolumeSpecName "kube-api-access-9bcn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:46:58.473182 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.473153    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcn2\" (UniqueName: \"kubernetes.io/projected/fb46f3f8-c7cf-4844-ae15-d731ddef5dd4-kube-api-access-9bcn2\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:46:58.710727 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.710658    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7818-8edad1"]
Sep 10 14:46:58.714462 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.714433    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7818-8edad1"]
Sep 10 14:46:58.831338 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.831310    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7818-5e031c"]
Sep 10 14:46:58.831648 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.831632    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e221f3-6f1a-466a-b671-7ceb103d1237" containerName="test-curl-7818-8edad1"
Sep 10 14:46:58.831704 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.831650    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e221f3-6f1a-466a-b671-7ceb103d1237" containerName="test-curl-7818-8edad1"
Sep 10 14:46:58.831758 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.831747    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e221f3-6f1a-466a-b671-7ceb103d1237" containerName="test-curl-7818-8edad1"
Sep 10 14:46:58.834636 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.834620    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:46:58.845192 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.845139    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7818-5e031c"]
Sep 10 14:46:58.876313 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.876285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw994\" (UniqueName: \"kubernetes.io/projected/47f07ea3-045f-4698-9ddb-af106485abbc-kube-api-access-vw994\") pod \"test-curl-7818-5e031c\" (UID: \"47f07ea3-045f-4698-9ddb-af106485abbc\") " pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:46:58.976777 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.976681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw994\" (UniqueName: \"kubernetes.io/projected/47f07ea3-045f-4698-9ddb-af106485abbc-kube-api-access-vw994\") pod \"test-curl-7818-5e031c\" (UID: \"47f07ea3-045f-4698-9ddb-af106485abbc\") " pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:46:58.984451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:58.984420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw994\" (UniqueName: \"kubernetes.io/projected/47f07ea3-045f-4698-9ddb-af106485abbc-kube-api-access-vw994\") pod \"test-curl-7818-5e031c\" (UID: \"47f07ea3-045f-4698-9ddb-af106485abbc\") " pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:46:59.128623 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:59.128593    2566 generic.go:356] "Generic (PLEG): container finished" podID="fb46f3f8-c7cf-4844-ae15-d731ddef5dd4" containerID="c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b" exitCode=137
Sep 10 14:46:59.129044 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:59.128636    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-0cd35a"
Sep 10 14:46:59.129044 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:59.128707    2566 scope.go:117] "RemoveContainer" containerID="c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b"
Sep 10 14:46:59.129044 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:59.128911    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7818-8edad1" container="test-curl-7818-8edad1"
Sep 10 14:46:59.137114 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:59.137097    2566 scope.go:117] "RemoveContainer" containerID="c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b"
Sep 10 14:46:59.137377 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:59.137359    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b\": container with ID starting with c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b not found: ID does not exist" containerID="c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b"
Sep 10 14:46:59.137432 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:59.137384    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b"} err="failed to get container status \"c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b\": rpc error: code = NotFound desc = could not find container \"c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b\": container with ID starting with c967a83ad6efb03fbe04bcae01598fb5935b39d98d7e6dc300f73e652dba712b not found: ID does not exist"
Sep 10 14:46:59.139090 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:46:59.139064    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7818-0cd35a\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="fb46f3f8-c7cf-4844-ae15-d731ddef5dd4" pod="opendatahub/test-curl-7818-0cd35a"
Sep 10 14:46:59.143234 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:59.143220    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:46:59.410670 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:46:59.410525    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7818-5e031c"]
Sep 10 14:46:59.413013 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:46:59.412983    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47f07ea3_045f_4698_9ddb_af106485abbc.slice/crio-968d0e939cae6adb1a35e5c1894cfae40fd91a35ddbae849a0cd4dd9e83297ea WatchSource:0}: Error finding container 968d0e939cae6adb1a35e5c1894cfae40fd91a35ddbae849a0cd4dd9e83297ea: Status 404 returned error can't find the container with id 968d0e939cae6adb1a35e5c1894cfae40fd91a35ddbae849a0cd4dd9e83297ea
Sep 10 14:47:00.110376 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.110344    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb46f3f8-c7cf-4844-ae15-d731ddef5dd4" path="/var/lib/kubelet/pods/fb46f3f8-c7cf-4844-ae15-d731ddef5dd4/volumes"
Sep 10 14:47:00.134946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.134920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7818-5e031c" event={"ID":"47f07ea3-045f-4698-9ddb-af106485abbc","Type":"ContainerStarted","Data":"f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8"}
Sep 10 14:47:00.135303 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.134958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7818-5e031c" event={"ID":"47f07ea3-045f-4698-9ddb-af106485abbc","Type":"ContainerStarted","Data":"968d0e939cae6adb1a35e5c1894cfae40fd91a35ddbae849a0cd4dd9e83297ea"}
Sep 10 14:47:00.136002 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.135973    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7818-8edad1" podUID="e2e221f3-6f1a-466a-b671-7ceb103d1237" containerName="test-curl-7818-8edad1" containerID="cri-o://ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5" gracePeriod=2
Sep 10 14:47:00.148732 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.148665    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7818-5e031c" podStartSLOduration=2.148642096 podStartE2EDuration="2.148642096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:00.147585503 +0000 UTC m=+1146.573322941" watchObservedRunningTime="2026-09-10 14:47:00.148642096 +0000 UTC m=+1146.574379514"
Sep 10 14:47:00.293492 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.293470    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-05a88a"
Sep 10 14:47:00.388997 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.388925    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkbvj\" (UniqueName: \"kubernetes.io/projected/c190b1a1-4a24-4b21-baec-97aca8181db7-kube-api-access-qkbvj\") pod \"c190b1a1-4a24-4b21-baec-97aca8181db7\" (UID: \"c190b1a1-4a24-4b21-baec-97aca8181db7\") "
Sep 10 14:47:00.390660 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.390628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c190b1a1-4a24-4b21-baec-97aca8181db7-kube-api-access-qkbvj" (OuterVolumeSpecName: "kube-api-access-qkbvj") pod "c190b1a1-4a24-4b21-baec-97aca8181db7" (UID: "c190b1a1-4a24-4b21-baec-97aca8181db7"). InnerVolumeSpecName "kube-api-access-qkbvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:00.489555 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.489526    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkbvj\" (UniqueName: \"kubernetes.io/projected/c190b1a1-4a24-4b21-baec-97aca8181db7-kube-api-access-qkbvj\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:00.780782 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.780719    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7818-5e031c"]
Sep 10 14:47:00.782560 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:00.782534    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7818-5e031c"]
Sep 10 14:47:01.141824 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:01.141739    2566 generic.go:356] "Generic (PLEG): container finished" podID="c190b1a1-4a24-4b21-baec-97aca8181db7" containerID="b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee" exitCode=137
Sep 10 14:47:01.141824 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:01.141789    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-05a88a"
Sep 10 14:47:01.142254 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:01.141830    2566 scope.go:117] "RemoveContainer" containerID="b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee"
Sep 10 14:47:01.142254 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:01.142045    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7818-5e031c" container="test-curl-7818-5e031c"
Sep 10 14:47:01.150441 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:01.150421    2566 scope.go:117] "RemoveContainer" containerID="b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee"
Sep 10 14:47:01.150742 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:01.150720    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee\": container with ID starting with b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee not found: ID does not exist" containerID="b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee"
Sep 10 14:47:01.150837 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:01.150750    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee"} err="failed to get container status \"b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee\": rpc error: code = NotFound desc = could not find container \"b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee\": container with ID starting with b38fbff600e7d3b00cb606086dc096b3fb8796823e5dfe4fb39542f55ca58eee not found: ID does not exist"
Sep 10 14:47:01.151658 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:01.151632    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7818-05a88a\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="c190b1a1-4a24-4b21-baec-97aca8181db7" pod="opendatahub/test-curl-7818-05a88a"
Sep 10 14:47:02.110141 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:02.110102    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c190b1a1-4a24-4b21-baec-97aca8181db7" path="/var/lib/kubelet/pods/c190b1a1-4a24-4b21-baec-97aca8181db7/volumes"
Sep 10 14:47:02.147089 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:02.147059    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7818-5e031c" podUID="47f07ea3-045f-4698-9ddb-af106485abbc" containerName="test-curl-7818-5e031c" containerID="cri-o://f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8" gracePeriod=2
Sep 10 14:47:02.319969 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:02.319947    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:47:02.405735 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:02.405683    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxzmp\" (UniqueName: \"kubernetes.io/projected/e2e221f3-6f1a-466a-b671-7ceb103d1237-kube-api-access-lxzmp\") pod \"e2e221f3-6f1a-466a-b671-7ceb103d1237\" (UID: \"e2e221f3-6f1a-466a-b671-7ceb103d1237\") "
Sep 10 14:47:02.407630 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:02.407604    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e221f3-6f1a-466a-b671-7ceb103d1237-kube-api-access-lxzmp" (OuterVolumeSpecName: "kube-api-access-lxzmp") pod "e2e221f3-6f1a-466a-b671-7ceb103d1237" (UID: "e2e221f3-6f1a-466a-b671-7ceb103d1237"). InnerVolumeSpecName "kube-api-access-lxzmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:02.506861 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:02.506830    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxzmp\" (UniqueName: \"kubernetes.io/projected/e2e221f3-6f1a-466a-b671-7ceb103d1237-kube-api-access-lxzmp\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:03.152508 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:03.152475    2566 generic.go:356] "Generic (PLEG): container finished" podID="e2e221f3-6f1a-466a-b671-7ceb103d1237" containerID="ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5" exitCode=137
Sep 10 14:47:03.152951 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:03.152516    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:47:03.152951 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:03.152576    2566 scope.go:117] "RemoveContainer" containerID="ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5"
Sep 10 14:47:03.161138 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:03.161122    2566 scope.go:117] "RemoveContainer" containerID="ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5"
Sep 10 14:47:03.161389 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:03.161373    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5\": container with ID starting with ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5 not found: ID does not exist" containerID="ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5"
Sep 10 14:47:03.161438 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:03.161397    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5"} err="failed to get container status \"ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5\": rpc error: code = NotFound desc = could not find container \"ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5\": container with ID starting with ee099dc5f3dd62e4b68659e5bd48abb94a42f6557a63aed8d459ad52cd1d76f5 not found: ID does not exist"
Sep 10 14:47:03.162470 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:03.162443    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7818-8edad1\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="e2e221f3-6f1a-466a-b671-7ceb103d1237" pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:47:04.108866 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:04.108824    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7818-8edad1\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="e2e221f3-6f1a-466a-b671-7ceb103d1237" pod="opendatahub/test-curl-7818-8edad1"
Sep 10 14:47:04.110373 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:04.110349    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e221f3-6f1a-466a-b671-7ceb103d1237" path="/var/lib/kubelet/pods/e2e221f3-6f1a-466a-b671-7ceb103d1237/volumes"
Sep 10 14:47:04.333476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:04.333449    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:47:04.422583 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:04.422561    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw994\" (UniqueName: \"kubernetes.io/projected/47f07ea3-045f-4698-9ddb-af106485abbc-kube-api-access-vw994\") pod \"47f07ea3-045f-4698-9ddb-af106485abbc\" (UID: \"47f07ea3-045f-4698-9ddb-af106485abbc\") "
Sep 10 14:47:04.424517 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:04.424492    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f07ea3-045f-4698-9ddb-af106485abbc-kube-api-access-vw994" (OuterVolumeSpecName: "kube-api-access-vw994") pod "47f07ea3-045f-4698-9ddb-af106485abbc" (UID: "47f07ea3-045f-4698-9ddb-af106485abbc"). InnerVolumeSpecName "kube-api-access-vw994". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:04.523479 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:04.523455    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw994\" (UniqueName: \"kubernetes.io/projected/47f07ea3-045f-4698-9ddb-af106485abbc-kube-api-access-vw994\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:05.164866 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:05.164829    2566 generic.go:356] "Generic (PLEG): container finished" podID="47f07ea3-045f-4698-9ddb-af106485abbc" containerID="f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8" exitCode=137
Sep 10 14:47:05.165042 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:05.164897    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:47:05.165042 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:05.164920    2566 scope.go:117] "RemoveContainer" containerID="f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8"
Sep 10 14:47:05.172942 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:05.172926    2566 scope.go:117] "RemoveContainer" containerID="f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8"
Sep 10 14:47:05.173213 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:05.173193    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8\": container with ID starting with f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8 not found: ID does not exist" containerID="f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8"
Sep 10 14:47:05.173345 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:05.173224    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8"} err="failed to get container status \"f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8\": rpc error: code = NotFound desc = could not find container \"f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8\": container with ID starting with f5a1cbb664adffec841ad53159dfd7241a3ad24498acee2faa019f7ac03c1bb8 not found: ID does not exist"
Sep 10 14:47:05.174433 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:05.174412    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7818-5e031c\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="47f07ea3-045f-4698-9ddb-af106485abbc" pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:47:06.100418 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.100389    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"]
Sep 10 14:47:06.100818 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.100770    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47f07ea3-045f-4698-9ddb-af106485abbc" containerName="test-curl-7818-5e031c"
Sep 10 14:47:06.100818 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.100786    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="47f07ea3-045f-4698-9ddb-af106485abbc" containerName="test-curl-7818-5e031c"
Sep 10 14:47:06.100896 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.100838    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="47f07ea3-045f-4698-9ddb-af106485abbc" containerName="test-curl-7818-5e031c"
Sep 10 14:47:06.105175 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.105154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.107881 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.107857    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e43b34d1-serving-cert\""
Sep 10 14:47:06.111158 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.111138    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47f07ea3-045f-4698-9ddb-af106485abbc" path="/var/lib/kubelet/pods/47f07ea3-045f-4698-9ddb-af106485abbc/volumes"
Sep 10 14:47:06.116917 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.116897    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"]
Sep 10 14:47:06.131968 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:06.131935    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7818-5e031c\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="47f07ea3-045f-4698-9ddb-af106485abbc" pod="opendatahub/test-curl-7818-5e031c"
Sep 10 14:47:06.240512 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.240476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2k8\" (UniqueName: \"kubernetes.io/projected/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-kube-api-access-rf2k8\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.240686 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.240551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-metrics-certs\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.240686 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.240590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-openshift-service-ca\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.240686 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.240678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-maas-api-tls\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.342219 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.342191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-metrics-certs\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.342219 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.342222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-openshift-service-ca\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.342450 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.342275    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-maas-api-tls\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.342450 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.342309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2k8\" (UniqueName: \"kubernetes.io/projected/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-kube-api-access-rf2k8\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.342996 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.342972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-openshift-service-ca\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.344607 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.344586    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-metrics-certs\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.344710 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.344635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-maas-api-tls\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.352806 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.352754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2k8\" (UniqueName: \"kubernetes.io/projected/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-kube-api-access-rf2k8\") pod \"maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.415260 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.415235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:06.687163 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.687137    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"]
Sep 10 14:47:06.689187 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:47:06.689149    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdf4fd2_ee0b_40fb_95e6_e6317a202bf7.slice/crio-bab7a8fcd38c98b8d9204325b62ada34531fb859c413c1f6c05339abb64ad1b1 WatchSource:0}: Error finding container bab7a8fcd38c98b8d9204325b62ada34531fb859c413c1f6c05339abb64ad1b1: Status 404 returned error can't find the container with id bab7a8fcd38c98b8d9204325b62ada34531fb859c413c1f6c05339abb64ad1b1
Sep 10 14:47:06.703807 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.703779    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"]
Sep 10 14:47:06.708232 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.708216    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:06.710400 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.710382    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e43b34d1\""
Sep 10 14:47:06.710507 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.710492    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e43b34d1-dockercfg-p5cbb\""
Sep 10 14:47:06.718314 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.718245    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"]
Sep 10 14:47:06.732604 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.732579    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"]
Sep 10 14:47:06.735848 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.735829    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:06.747813 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.747789    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"]
Sep 10 14:47:06.846862 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.846819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db00bb3e-ae47-48b0-b296-54f92aca5c07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc\" (UID: \"db00bb3e-ae47-48b0-b296-54f92aca5c07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:06.847057 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.846902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e6bd0e-f151-46cd-95ee-b091437e1d0a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l\" (UID: \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\") " pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:06.847182 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.847111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznwc\" (UniqueName: \"kubernetes.io/projected/db00bb3e-ae47-48b0-b296-54f92aca5c07-kube-api-access-hznwc\") pod \"payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc\" (UID: \"db00bb3e-ae47-48b0-b296-54f92aca5c07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:06.847240 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.847207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hl62\" (UniqueName: \"kubernetes.io/projected/12e6bd0e-f151-46cd-95ee-b091437e1d0a-kube-api-access-6hl62\") pod \"payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l\" (UID: \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\") " pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:06.948554 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.948478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db00bb3e-ae47-48b0-b296-54f92aca5c07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc\" (UID: \"db00bb3e-ae47-48b0-b296-54f92aca5c07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:06.948554 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.948530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e6bd0e-f151-46cd-95ee-b091437e1d0a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l\" (UID: \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\") " pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:06.948764 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.948569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznwc\" (UniqueName: \"kubernetes.io/projected/db00bb3e-ae47-48b0-b296-54f92aca5c07-kube-api-access-hznwc\") pod \"payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc\" (UID: \"db00bb3e-ae47-48b0-b296-54f92aca5c07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:06.948764 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.948593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hl62\" (UniqueName: \"kubernetes.io/projected/12e6bd0e-f151-46cd-95ee-b091437e1d0a-kube-api-access-6hl62\") pod \"payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l\" (UID: \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\") " pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:06.949184 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.949163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db00bb3e-ae47-48b0-b296-54f92aca5c07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc\" (UID: \"db00bb3e-ae47-48b0-b296-54f92aca5c07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:06.949297 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.949279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e6bd0e-f151-46cd-95ee-b091437e1d0a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l\" (UID: \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\") " pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:06.956001 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.955975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznwc\" (UniqueName: \"kubernetes.io/projected/db00bb3e-ae47-48b0-b296-54f92aca5c07-kube-api-access-hznwc\") pod \"payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc\" (UID: \"db00bb3e-ae47-48b0-b296-54f92aca5c07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:06.956103 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:06.956037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hl62\" (UniqueName: \"kubernetes.io/projected/12e6bd0e-f151-46cd-95ee-b091437e1d0a-kube-api-access-6hl62\") pod \"payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l\" (UID: \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\") " pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:07.025037 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:07.025006    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:07.045057 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:07.045030    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:07.176039 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:07.176004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9" event={"ID":"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7","Type":"ContainerStarted","Data":"bab7a8fcd38c98b8d9204325b62ada34531fb859c413c1f6c05339abb64ad1b1"}
Sep 10 14:47:07.246207 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:07.246183    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:47:07.246299 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:07.246230    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:47:07.258877 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:07.258849    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:47:07.298226 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:07.298202    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"]
Sep 10 14:47:07.299462 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:47:07.299436    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb00bb3e_ae47_48b0_b296_54f92aca5c07.slice/crio-6284d282242e0c0b1d83ffda7057b2b10ae5ee11275bd880c4c19e0e93b83c1c WatchSource:0}: Error finding container 6284d282242e0c0b1d83ffda7057b2b10ae5ee11275bd880c4c19e0e93b83c1c: Status 404 returned error can't find the container with id 6284d282242e0c0b1d83ffda7057b2b10ae5ee11275bd880c4c19e0e93b83c1c
Sep 10 14:47:07.319666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:07.319638    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"]
Sep 10 14:47:07.322596 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:47:07.322573    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e6bd0e_f151_46cd_95ee_b091437e1d0a.slice/crio-c834ec66d2206da012578f5c92c5bebf1238417ed96cac909c214514a4711926 WatchSource:0}: Error finding container c834ec66d2206da012578f5c92c5bebf1238417ed96cac909c214514a4711926: Status 404 returned error can't find the container with id c834ec66d2206da012578f5c92c5bebf1238417ed96cac909c214514a4711926
Sep 10 14:47:08.182255 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.182220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l" event={"ID":"12e6bd0e-f151-46cd-95ee-b091437e1d0a","Type":"ContainerStarted","Data":"082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194"}
Sep 10 14:47:08.182675 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.182274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l" event={"ID":"12e6bd0e-f151-46cd-95ee-b091437e1d0a","Type":"ContainerStarted","Data":"c834ec66d2206da012578f5c92c5bebf1238417ed96cac909c214514a4711926"}
Sep 10 14:47:08.182675 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.182325    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:08.183779 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.183759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc" event={"ID":"db00bb3e-ae47-48b0-b296-54f92aca5c07","Type":"ContainerStarted","Data":"798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9"}
Sep 10 14:47:08.183846 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.183794    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc" event={"ID":"db00bb3e-ae47-48b0-b296-54f92aca5c07","Type":"ContainerStarted","Data":"6284d282242e0c0b1d83ffda7057b2b10ae5ee11275bd880c4c19e0e93b83c1c"}
Sep 10 14:47:08.183910 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.183853    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:08.185365 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.185345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9" event={"ID":"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7","Type":"ContainerStarted","Data":"d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51"}
Sep 10 14:47:08.185606 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.185587    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:08.196414 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.196397    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:47:08.199394 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.199353    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l" podStartSLOduration=2.199340703 podStartE2EDuration="2.199340703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:08.198453089 +0000 UTC m=+1154.624190519" watchObservedRunningTime="2026-09-10 14:47:08.199340703 +0000 UTC m=+1154.625078186"
Sep 10 14:47:08.215553 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.215509    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9" podStartSLOduration=1.706221319 podStartE2EDuration="2.215495467s" totalImagesPullingTime="509.274148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:08.215035085 +0000 UTC m=+1154.640772504" watchObservedRunningTime="2026-09-10 14:47:08.215495467 +0000 UTC m=+1154.641232886"
Sep 10 14:47:08.249909 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:08.249855    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc" podStartSLOduration=2.249834735 podStartE2EDuration="2.249834735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:08.247158157 +0000 UTC m=+1154.672895590" watchObservedRunningTime="2026-09-10 14:47:08.249834735 +0000 UTC m=+1154.675572157"
Sep 10 14:47:14.199106 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:14.199076    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:47:19.195652 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:19.195565    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:47:19.196515 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:19.196494    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:47:29.980936 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:29.980900    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"]
Sep 10 14:47:29.984397 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:29.984376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:29.986539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:29.986518    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-12b6f6-dockercfg-cr222\""
Sep 10 14:47:29.986625 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:29.986518    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-12b6f6\""
Sep 10 14:47:29.993113 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:29.993090    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"]
Sep 10 14:47:30.150787 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:30.150747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7fb\" (UniqueName: \"kubernetes.io/projected/e3e574fc-35a9-4acc-84f3-6b437bb03925-kube-api-access-mz7fb\") pod \"payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg\" (UID: \"e3e574fc-35a9-4acc-84f3-6b437bb03925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:30.150952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:30.150806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e574fc-35a9-4acc-84f3-6b437bb03925-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg\" (UID: \"e3e574fc-35a9-4acc-84f3-6b437bb03925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:30.251941 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:30.251867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7fb\" (UniqueName: \"kubernetes.io/projected/e3e574fc-35a9-4acc-84f3-6b437bb03925-kube-api-access-mz7fb\") pod \"payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg\" (UID: \"e3e574fc-35a9-4acc-84f3-6b437bb03925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:30.251941 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:30.251911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e574fc-35a9-4acc-84f3-6b437bb03925-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg\" (UID: \"e3e574fc-35a9-4acc-84f3-6b437bb03925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:30.252589 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:30.252568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e574fc-35a9-4acc-84f3-6b437bb03925-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg\" (UID: \"e3e574fc-35a9-4acc-84f3-6b437bb03925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:30.260316 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:30.260287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7fb\" (UniqueName: \"kubernetes.io/projected/e3e574fc-35a9-4acc-84f3-6b437bb03925-kube-api-access-mz7fb\") pod \"payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg\" (UID: \"e3e574fc-35a9-4acc-84f3-6b437bb03925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:30.295408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:30.295383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:30.563426 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:30.563389    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"]
Sep 10 14:47:30.565496 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:47:30.565462    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e574fc_35a9_4acc_84f3_6b437bb03925.slice/crio-f240c84e03c1f184c78af6ea64d5a59782956cc1657bab513f335417d408cbd0 WatchSource:0}: Error finding container f240c84e03c1f184c78af6ea64d5a59782956cc1657bab513f335417d408cbd0: Status 404 returned error can't find the container with id f240c84e03c1f184c78af6ea64d5a59782956cc1657bab513f335417d408cbd0
Sep 10 14:47:31.306855 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:31.306819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg" event={"ID":"e3e574fc-35a9-4acc-84f3-6b437bb03925","Type":"ContainerStarted","Data":"a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744"}
Sep 10 14:47:31.307386 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:31.306875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg" event={"ID":"e3e574fc-35a9-4acc-84f3-6b437bb03925","Type":"ContainerStarted","Data":"f240c84e03c1f184c78af6ea64d5a59782956cc1657bab513f335417d408cbd0"}
Sep 10 14:47:31.307386 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:31.306951    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:31.323211 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:31.323163    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg" podStartSLOduration=2.323145584 podStartE2EDuration="2.323145584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:31.32103815 +0000 UTC m=+1177.746775569" watchObservedRunningTime="2026-09-10 14:47:31.323145584 +0000 UTC m=+1177.748883005"
Sep 10 14:47:34.842895 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:34.842862    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"]
Sep 10 14:47:42.313148 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:42.313115    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 14:47:49.862753 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:49.862687    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk" podUID="e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" containerName="main" containerID="cri-o://e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746" gracePeriod=45
Sep 10 14:47:50.112966 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.112907    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:47:50.195022 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.194988    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-model-cache\") pod \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") "
Sep 10 14:47:50.195191 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195035    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-home\") pod \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") "
Sep 10 14:47:50.195191 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tls-certs\") pod \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") "
Sep 10 14:47:50.195191 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195120    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf8fc\" (UniqueName: \"kubernetes.io/projected/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-kube-api-access-wf8fc\") pod \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") "
Sep 10 14:47:50.195191 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195158    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-dshm\") pod \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") "
Sep 10 14:47:50.195428 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195213    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tmp-dir\") pod \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\" (UID: \"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e\") "
Sep 10 14:47:50.195428 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195312    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" (UID: "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:50.195428 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195364    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-home" (OuterVolumeSpecName: "home") pod "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" (UID: "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:50.195593 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195529    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:50.195593 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195545    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:50.195593 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.195544    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" (UID: "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:50.197320 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.197281    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-dshm" (OuterVolumeSpecName: "dshm") pod "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" (UID: "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:50.197433 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.197328    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" (UID: "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:50.197433 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.197403    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-kube-api-access-wf8fc" (OuterVolumeSpecName: "kube-api-access-wf8fc") pod "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" (UID: "e378fa4d-e0f9-4d3c-9d06-65ada3704e5e"). InnerVolumeSpecName "kube-api-access-wf8fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:50.296183 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.296155    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:50.296183 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.296179    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:50.296324 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.296191    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8fc\" (UniqueName: \"kubernetes.io/projected/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-kube-api-access-wf8fc\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:50.296324 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.296203    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:47:50.404477 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.404446    2566 generic.go:356] "Generic (PLEG): container finished" podID="e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" containerID="e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746" exitCode=0
Sep 10 14:47:50.404619 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.404489    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"
Sep 10 14:47:50.404619 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.404535    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk" event={"ID":"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e","Type":"ContainerDied","Data":"e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746"}
Sep 10 14:47:50.404619 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.404584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk" event={"ID":"e378fa4d-e0f9-4d3c-9d06-65ada3704e5e","Type":"ContainerDied","Data":"07635ea1de56fa095577f8440322ff8b813e8593a2e4882a85d6588e676e8d3f"}
Sep 10 14:47:50.404619 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.404608    2566 scope.go:117] "RemoveContainer" containerID="e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746"
Sep 10 14:47:50.416792 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.416772    2566 scope.go:117] "RemoveContainer" containerID="e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746"
Sep 10 14:47:50.417066 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:50.417047    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746\": container with ID starting with e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746 not found: ID does not exist" containerID="e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746"
Sep 10 14:47:50.417132 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.417079    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746"} err="failed to get container status \"e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746\": rpc error: code = NotFound desc = could not find container \"e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746\": container with ID starting with e404f7e5e57d4e4be17e4aad65af2f125cd7ae1aed0121b35f593d91f220c746 not found: ID does not exist"
Sep 10 14:47:50.426358 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.426335    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"]
Sep 10 14:47:50.429722 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:50.429684    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-b20fba/noten-model-b20fba-kserve-5c884cf555-5lhtk"]
Sep 10 14:47:52.109917 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:52.109883    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" path="/var/lib/kubelet/pods/e378fa4d-e0f9-4d3c-9d06-65ada3704e5e/volumes"
Sep 10 14:47:54.051193 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:54.051161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:47:54.052109 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:54.052090    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:47:54.055148 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:54.055132    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:47:54.055877 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:54.055861    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:47:58.557308 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.557271    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"]
Sep 10 14:47:58.557649 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.557613    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" containerName="main"
Sep 10 14:47:58.557649 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.557625    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" containerName="main"
Sep 10 14:47:58.557750 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.557708    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e378fa4d-e0f9-4d3c-9d06-65ada3704e5e" containerName="main"
Sep 10 14:47:58.562175 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.562150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.564478 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.564456    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e6941c-serving-cert\""
Sep 10 14:47:58.574914 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.574894    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"]
Sep 10 14:47:58.662374 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.662343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32d6e0-b1da-421b-9f36-d714627a81aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.662546 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.662385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.662546 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.662415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.662546 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.662448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnq5m\" (UniqueName: \"kubernetes.io/projected/9c32d6e0-b1da-421b-9f36-d714627a81aa-kube-api-access-tnq5m\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.763373 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.763342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32d6e0-b1da-421b-9f36-d714627a81aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.763509 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.763382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.763509 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.763399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.763509 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.763424    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnq5m\" (UniqueName: \"kubernetes.io/projected/9c32d6e0-b1da-421b-9f36-d714627a81aa-kube-api-access-tnq5m\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.763509 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:58.763494    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-serving-cert: secret "maas-api-e2e-shared-b-w3-e6941c-serving-cert" not found
Sep 10 14:47:58.763652 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:47:58.763555    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls podName:9c32d6e0-b1da-421b-9f36-d714627a81aa nodeName:}" failed. No retries permitted until 2026-09-10 14:47:59.263540267 +0000 UTC m=+1205.689277664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w" (UID: "9c32d6e0-b1da-421b-9f36-d714627a81aa") : secret "maas-api-e2e-shared-b-w3-e6941c-serving-cert" not found
Sep 10 14:47:58.764158 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.764140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32d6e0-b1da-421b-9f36-d714627a81aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.765869 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.765851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:58.771436 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:58.771412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnq5m\" (UniqueName: \"kubernetes.io/projected/9c32d6e0-b1da-421b-9f36-d714627a81aa-kube-api-access-tnq5m\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:59.267766 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:59.267736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:59.269970 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:59.269943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:59.472668 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:59.472635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:47:59.762249 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:47:59.762223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"]
Sep 10 14:47:59.764060 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:47:59.764034    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c32d6e0_b1da_421b_9f36_d714627a81aa.slice/crio-9fafb7e0f8a034b152c08e8d39a6aaf28a356d6ccb39dce3d6634d64ee9cd22d WatchSource:0}: Error finding container 9fafb7e0f8a034b152c08e8d39a6aaf28a356d6ccb39dce3d6634d64ee9cd22d: Status 404 returned error can't find the container with id 9fafb7e0f8a034b152c08e8d39a6aaf28a356d6ccb39dce3d6634d64ee9cd22d
Sep 10 14:48:00.450219 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:00.450133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w" event={"ID":"9c32d6e0-b1da-421b-9f36-d714627a81aa","Type":"ContainerStarted","Data":"d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f"}
Sep 10 14:48:00.450219 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:00.450191    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w" event={"ID":"9c32d6e0-b1da-421b-9f36-d714627a81aa","Type":"ContainerStarted","Data":"9fafb7e0f8a034b152c08e8d39a6aaf28a356d6ccb39dce3d6634d64ee9cd22d"}
Sep 10 14:48:00.450408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:00.450252    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:48:00.468079 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:00.468037    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w" podStartSLOduration=2.039337353 podStartE2EDuration="2.468024066s" totalImagesPullingTime="428.686713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:00.467211276 +0000 UTC m=+1206.892948694" watchObservedRunningTime="2026-09-10 14:48:00.468024066 +0000 UTC m=+1206.893761485"
Sep 10 14:48:06.466476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:06.466443    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 14:48:17.734670 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.734637    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"]
Sep 10 14:48:17.738956 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.738941    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.741218 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.741196    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"unconfigured-w4-5b0323-kserve-self-signed-certs\""
Sep 10 14:48:17.748751 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.748729    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"]
Sep 10 14:48:17.821497 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.821464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tls-certs\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.821656 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.821512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-dshm\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.821656 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.821537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-home\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.821656 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.821564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-model-cache\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.821656 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.821611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rsd\" (UniqueName: \"kubernetes.io/projected/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-kube-api-access-44rsd\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.821656 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.821654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tmp-dir\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.922622 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.922593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tls-certs\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.922763 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.922627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-dshm\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.922763 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.922642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-home\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.922763 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.922661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-model-cache\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.922763 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.922728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rsd\" (UniqueName: \"kubernetes.io/projected/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-kube-api-access-44rsd\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.922763 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.922754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tmp-dir\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.923009 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.922959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-home\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.923117 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.923093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-model-cache\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.923233 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.923127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tmp-dir\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.924705 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.924677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-dshm\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.925174 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.925152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tls-certs\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:17.935162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:17.935120    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rsd\" (UniqueName: \"kubernetes.io/projected/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-kube-api-access-44rsd\") pod \"unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") " pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:18.048130 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:18.048050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:18.320779 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:18.320756    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"]
Sep 10 14:48:18.322561 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:48:18.322533    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fe3c3d2_3b86_4e52_91d3_a47afd61b93d.slice/crio-980c122eaa547f00e44da2cf5d529011a750d06334ee9b79a169cc0b9b0e8350 WatchSource:0}: Error finding container 980c122eaa547f00e44da2cf5d529011a750d06334ee9b79a169cc0b9b0e8350: Status 404 returned error can't find the container with id 980c122eaa547f00e44da2cf5d529011a750d06334ee9b79a169cc0b9b0e8350
Sep 10 14:48:18.538279 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:18.538248    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll" event={"ID":"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d","Type":"ContainerStarted","Data":"9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a"}
Sep 10 14:48:18.538434 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:18.538295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll" event={"ID":"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d","Type":"ContainerStarted","Data":"980c122eaa547f00e44da2cf5d529011a750d06334ee9b79a169cc0b9b0e8350"}
Sep 10 14:48:18.555635 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:18.555594    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll" podStartSLOduration=1.55558141 podStartE2EDuration="1.55558141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:18.554162284 +0000 UTC m=+1224.979899702" watchObservedRunningTime="2026-09-10 14:48:18.55558141 +0000 UTC m=+1224.981318828"
Sep 10 14:48:28.048642 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:28.048608    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:28.049154 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:28.048666    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:28.060428 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:28.060405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:48:28.599524 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:48:28.599499    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:49:17.045992 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.045914    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"]
Sep 10 14:49:17.049579 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.049556    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.052741 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.052716    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"openshift-service-ca.crt\""
Sep 10 14:49:17.052741 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.052723    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"auth-test-model-12b6f6-kserve-self-signed-certs\""
Sep 10 14:49:17.052741 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.052737    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"kube-root-ca.crt\""
Sep 10 14:49:17.053001 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.052768    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"default-dockercfg-lrmnz\""
Sep 10 14:49:17.057368 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.057347    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"]
Sep 10 14:49:17.099722 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.099678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tmp-dir\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.099838 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.099730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-home\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.099909 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.099861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tls-certs\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.099909 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.099898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-dshm\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.100016 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.099979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-model-cache\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.100079 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.100025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgmr\" (UniqueName: \"kubernetes.io/projected/f3ee621e-9a06-49eb-aa97-4deed322e6d0-kube-api-access-kfgmr\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201275 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tmp-dir\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-home\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tls-certs\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-dshm\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-model-cache\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201466 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgmr\" (UniqueName: \"kubernetes.io/projected/f3ee621e-9a06-49eb-aa97-4deed322e6d0-kube-api-access-kfgmr\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201715    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tmp-dir\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201774 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-home\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.201883 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.201857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-model-cache\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.203998 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.203975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-dshm\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.204344 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.204322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tls-certs\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.209531 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.209503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgmr\" (UniqueName: \"kubernetes.io/projected/f3ee621e-9a06-49eb-aa97-4deed322e6d0-kube-api-access-kfgmr\") pod \"auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") " pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.362106 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.362021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:17.636661 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.636634    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"]
Sep 10 14:49:17.638762 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:49:17.638733    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3ee621e_9a06_49eb_aa97_4deed322e6d0.slice/crio-5b3cf551755f3c8325af386553688c9ade12cc8d6070c4c10f059f31ac02a572 WatchSource:0}: Error finding container 5b3cf551755f3c8325af386553688c9ade12cc8d6070c4c10f059f31ac02a572: Status 404 returned error can't find the container with id 5b3cf551755f3c8325af386553688c9ade12cc8d6070c4c10f059f31ac02a572
Sep 10 14:49:17.822297 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.822262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj" event={"ID":"f3ee621e-9a06-49eb-aa97-4deed322e6d0","Type":"ContainerStarted","Data":"5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e"}
Sep 10 14:49:17.822297 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.822301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj" event={"ID":"f3ee621e-9a06-49eb-aa97-4deed322e6d0","Type":"ContainerStarted","Data":"5b3cf551755f3c8325af386553688c9ade12cc8d6070c4c10f059f31ac02a572"}
Sep 10 14:49:17.838372 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:17.838318    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj" podStartSLOduration=0.838301338 podStartE2EDuration="838.301338ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:17.837926166 +0000 UTC m=+1284.263663588" watchObservedRunningTime="2026-09-10 14:49:17.838301338 +0000 UTC m=+1284.264038758"
Sep 10 14:49:27.362554 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:27.362517    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:27.363059 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:27.362583    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:27.375138 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:27.375113    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:27.882440 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:27.882413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:49:38.464928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.464895    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"]
Sep 10 14:49:38.465346 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.465105    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9" podUID="3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" containerName="maas-api" containerID="cri-o://d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51" gracePeriod=30
Sep 10 14:49:38.545068 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.544897    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"]
Sep 10 14:49:38.545260 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.545178    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l" podUID="12e6bd0e-f151-46cd-95ee-b091437e1d0a" containerName="payload-processing" containerID="cri-o://082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194" gracePeriod=30
Sep 10 14:49:38.557509 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.557470    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"]
Sep 10 14:49:38.557890 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.557770    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc" podUID="db00bb3e-ae47-48b0-b296-54f92aca5c07" containerName="payload-pre-processing" containerID="cri-o://798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9" gracePeriod=30
Sep 10 14:49:38.861284 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.861258    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:49:38.865477 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.865448    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:49:38.872854 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.872829    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:49:38.924820 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.924778    2566 generic.go:356] "Generic (PLEG): container finished" podID="3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" containerID="d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51" exitCode=0
Sep 10 14:49:38.925005 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.924833    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"
Sep 10 14:49:38.925005 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.924861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9" event={"ID":"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7","Type":"ContainerDied","Data":"d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51"}
Sep 10 14:49:38.925005 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.924913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9" event={"ID":"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7","Type":"ContainerDied","Data":"bab7a8fcd38c98b8d9204325b62ada34531fb859c413c1f6c05339abb64ad1b1"}
Sep 10 14:49:38.925005 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.924937    2566 scope.go:117] "RemoveContainer" containerID="d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51"
Sep 10 14:49:38.926346 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.926325    2566 generic.go:356] "Generic (PLEG): container finished" podID="12e6bd0e-f151-46cd-95ee-b091437e1d0a" containerID="082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194" exitCode=0
Sep 10 14:49:38.926437 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.926364    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"
Sep 10 14:49:38.926437 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.926406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l" event={"ID":"12e6bd0e-f151-46cd-95ee-b091437e1d0a","Type":"ContainerDied","Data":"082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194"}
Sep 10 14:49:38.926558 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.926518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l" event={"ID":"12e6bd0e-f151-46cd-95ee-b091437e1d0a","Type":"ContainerDied","Data":"c834ec66d2206da012578f5c92c5bebf1238417ed96cac909c214514a4711926"}
Sep 10 14:49:38.927791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.927758    2566 generic.go:356] "Generic (PLEG): container finished" podID="db00bb3e-ae47-48b0-b296-54f92aca5c07" containerID="798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9" exitCode=0
Sep 10 14:49:38.927791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.927778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc" event={"ID":"db00bb3e-ae47-48b0-b296-54f92aca5c07","Type":"ContainerDied","Data":"798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9"}
Sep 10 14:49:38.927924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.927798    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"
Sep 10 14:49:38.927924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.927824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc" event={"ID":"db00bb3e-ae47-48b0-b296-54f92aca5c07","Type":"ContainerDied","Data":"6284d282242e0c0b1d83ffda7057b2b10ae5ee11275bd880c4c19e0e93b83c1c"}
Sep 10 14:49:38.934323 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.934300    2566 scope.go:117] "RemoveContainer" containerID="d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51"
Sep 10 14:49:38.934640 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:49:38.934614    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51\": container with ID starting with d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51 not found: ID does not exist" containerID="d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51"
Sep 10 14:49:38.934750 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.934641    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51"} err="failed to get container status \"d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51\": rpc error: code = NotFound desc = could not find container \"d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51\": container with ID starting with d7b70f036c656e3bc6d641be0ab53dee2384051da3a6b14bbfe94ccd82ecfd51 not found: ID does not exist"
Sep 10 14:49:38.934750 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.934657    2566 scope.go:117] "RemoveContainer" containerID="082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194"
Sep 10 14:49:38.942438 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.942420    2566 scope.go:117] "RemoveContainer" containerID="082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194"
Sep 10 14:49:38.942710 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:49:38.942677    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194\": container with ID starting with 082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194 not found: ID does not exist" containerID="082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194"
Sep 10 14:49:38.942759 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.942721    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194"} err="failed to get container status \"082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194\": rpc error: code = NotFound desc = could not find container \"082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194\": container with ID starting with 082e77944d2a5d74e8258973ff05bd27b663ede2837ca81e33f8326803d9d194 not found: ID does not exist"
Sep 10 14:49:38.942759 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.942737    2566 scope.go:117] "RemoveContainer" containerID="798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9"
Sep 10 14:49:38.950090 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.950073    2566 scope.go:117] "RemoveContainer" containerID="798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9"
Sep 10 14:49:38.950332 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:49:38.950316    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9\": container with ID starting with 798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9 not found: ID does not exist" containerID="798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9"
Sep 10 14:49:38.950394 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.950338    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9"} err="failed to get container status \"798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9\": rpc error: code = NotFound desc = could not find container \"798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9\": container with ID starting with 798e10817e4f0388da70a7ae46514990374828cc9a2fceb5f7d26c90a3ebdff9 not found: ID does not exist"
Sep 10 14:49:38.993225 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hl62\" (UniqueName: \"kubernetes.io/projected/12e6bd0e-f151-46cd-95ee-b091437e1d0a-kube-api-access-6hl62\") pod \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\" (UID: \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\") "
Sep 10 14:49:38.993225 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993213    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-metrics-certs\") pod \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") "
Sep 10 14:49:38.993406 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993237    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hznwc\" (UniqueName: \"kubernetes.io/projected/db00bb3e-ae47-48b0-b296-54f92aca5c07-kube-api-access-hznwc\") pod \"db00bb3e-ae47-48b0-b296-54f92aca5c07\" (UID: \"db00bb3e-ae47-48b0-b296-54f92aca5c07\") "
Sep 10 14:49:38.993406 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993256    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-openshift-service-ca\") pod \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") "
Sep 10 14:49:38.993406 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993306    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-maas-api-tls\") pod \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") "
Sep 10 14:49:38.993406 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf2k8\" (UniqueName: \"kubernetes.io/projected/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-kube-api-access-rf2k8\") pod \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\" (UID: \"3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7\") "
Sep 10 14:49:38.993622 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993418    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db00bb3e-ae47-48b0-b296-54f92aca5c07-plugins-config-volume\") pod \"db00bb3e-ae47-48b0-b296-54f92aca5c07\" (UID: \"db00bb3e-ae47-48b0-b296-54f92aca5c07\") "
Sep 10 14:49:38.993622 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e6bd0e-f151-46cd-95ee-b091437e1d0a-plugins-config-volume\") pod \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\" (UID: \"12e6bd0e-f151-46cd-95ee-b091437e1d0a\") "
Sep 10 14:49:38.993776 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993652    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" (UID: "3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:49:38.993834 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.993817    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:49:38.994082 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.994041    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e6bd0e-f151-46cd-95ee-b091437e1d0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12e6bd0e-f151-46cd-95ee-b091437e1d0a" (UID: "12e6bd0e-f151-46cd-95ee-b091437e1d0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:49:38.994152 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.994111    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db00bb3e-ae47-48b0-b296-54f92aca5c07-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db00bb3e-ae47-48b0-b296-54f92aca5c07" (UID: "db00bb3e-ae47-48b0-b296-54f92aca5c07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:49:38.995509 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.995483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" (UID: "3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:49:38.995509 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.995498    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db00bb3e-ae47-48b0-b296-54f92aca5c07-kube-api-access-hznwc" (OuterVolumeSpecName: "kube-api-access-hznwc") pod "db00bb3e-ae47-48b0-b296-54f92aca5c07" (UID: "db00bb3e-ae47-48b0-b296-54f92aca5c07"). InnerVolumeSpecName "kube-api-access-hznwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:38.995635 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.995525    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e6bd0e-f151-46cd-95ee-b091437e1d0a-kube-api-access-6hl62" (OuterVolumeSpecName: "kube-api-access-6hl62") pod "12e6bd0e-f151-46cd-95ee-b091437e1d0a" (UID: "12e6bd0e-f151-46cd-95ee-b091437e1d0a"). InnerVolumeSpecName "kube-api-access-6hl62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:38.995824 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.995810    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" (UID: "3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:49:38.996109 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:38.996085    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-kube-api-access-rf2k8" (OuterVolumeSpecName: "kube-api-access-rf2k8") pod "3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" (UID: "3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7"). InnerVolumeSpecName "kube-api-access-rf2k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:39.094791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.094752    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:49:39.094791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.094786    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2k8\" (UniqueName: \"kubernetes.io/projected/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-kube-api-access-rf2k8\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:49:39.094791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.094797    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db00bb3e-ae47-48b0-b296-54f92aca5c07-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:49:39.095024 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.094805    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e6bd0e-f151-46cd-95ee-b091437e1d0a-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:49:39.095024 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.094814    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hl62\" (UniqueName: \"kubernetes.io/projected/12e6bd0e-f151-46cd-95ee-b091437e1d0a-kube-api-access-6hl62\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:49:39.095024 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.094822    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:49:39.095024 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.094830    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznwc\" (UniqueName: \"kubernetes.io/projected/db00bb3e-ae47-48b0-b296-54f92aca5c07-kube-api-access-hznwc\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:49:39.249609 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.249545    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"]
Sep 10 14:49:39.253171 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.253147    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e43b34d1-6dd99854f7-w2sl9"]
Sep 10 14:49:39.262389 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.262363    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"]
Sep 10 14:49:39.266726 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.266686    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43b34d1-f8ddd7bc7-96x6l"]
Sep 10 14:49:39.276606 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.276586    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"]
Sep 10 14:49:39.281451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:39.281429    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43b34d1-86c5f9d85d-l9xcc"]
Sep 10 14:49:40.110948 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:40.110910    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e6bd0e-f151-46cd-95ee-b091437e1d0a" path="/var/lib/kubelet/pods/12e6bd0e-f151-46cd-95ee-b091437e1d0a/volumes"
Sep 10 14:49:40.111346 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:40.111331    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" path="/var/lib/kubelet/pods/3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7/volumes"
Sep 10 14:49:40.111722 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:40.111682    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db00bb3e-ae47-48b0-b296-54f92aca5c07" path="/var/lib/kubelet/pods/db00bb3e-ae47-48b0-b296-54f92aca5c07/volumes"
Sep 10 14:49:53.433100 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433066    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"]
Sep 10 14:49:53.433559 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433540    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" containerName="maas-api"
Sep 10 14:49:53.433650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433563    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" containerName="maas-api"
Sep 10 14:49:53.433650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433592    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db00bb3e-ae47-48b0-b296-54f92aca5c07" containerName="payload-pre-processing"
Sep 10 14:49:53.433650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433601    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="db00bb3e-ae47-48b0-b296-54f92aca5c07" containerName="payload-pre-processing"
Sep 10 14:49:53.433650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433614    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e6bd0e-f151-46cd-95ee-b091437e1d0a" containerName="payload-processing"
Sep 10 14:49:53.433650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433623    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e6bd0e-f151-46cd-95ee-b091437e1d0a" containerName="payload-processing"
Sep 10 14:49:53.433823 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433719    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bdf4fd2-ee0b-40fb-95e6-e6317a202bf7" containerName="maas-api"
Sep 10 14:49:53.433823 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433729    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="db00bb3e-ae47-48b0-b296-54f92aca5c07" containerName="payload-pre-processing"
Sep 10 14:49:53.433823 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.433740    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e6bd0e-f151-46cd-95ee-b091437e1d0a" containerName="payload-processing"
Sep 10 14:49:53.436920 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.436898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.439204 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.439181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"embedding-w4-5b0323-kserve-self-signed-certs\""
Sep 10 14:49:53.443134 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.443109    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"]
Sep 10 14:49:53.608319 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.608278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txwt\" (UniqueName: \"kubernetes.io/projected/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-kube-api-access-5txwt\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.608319 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.608323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tls-certs\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.608562 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.608379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-dshm\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.608562 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.608486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tmp-dir\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.608562 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.608540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-model-cache\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.608732 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.608596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-home\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.709848 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.709780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-model-cache\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.709848 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.709816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-home\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.709848 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.709836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txwt\" (UniqueName: \"kubernetes.io/projected/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-kube-api-access-5txwt\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.710050 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.709872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tls-certs\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.710050 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.709906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-dshm\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.710050 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.709959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tmp-dir\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.710238 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.710215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-model-cache\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.710297 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.710272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-home\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.710297 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.710284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tmp-dir\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.712108 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.712085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-dshm\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.712261 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.712243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tls-certs\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.717317 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.717289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txwt\" (UniqueName: \"kubernetes.io/projected/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-kube-api-access-5txwt\") pod \"embedding-w4-5b0323-kserve-6f96646559-b7zxr\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") " pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:53.746337 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:53.746319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:49:54.013421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:54.013258    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"]
Sep 10 14:49:54.015926 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:49:54.015904    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ad8c06c_0bc0_4f7a_8a5c_b0f4479393e3.slice/crio-a105397454bde86398d2def21ab6576579028238dd6248edb81662fe072c67e2 WatchSource:0}: Error finding container a105397454bde86398d2def21ab6576579028238dd6248edb81662fe072c67e2: Status 404 returned error can't find the container with id a105397454bde86398d2def21ab6576579028238dd6248edb81662fe072c67e2
Sep 10 14:49:55.006433 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:55.006393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr" event={"ID":"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3","Type":"ContainerStarted","Data":"96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac"}
Sep 10 14:49:55.006934 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:55.006442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr" event={"ID":"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3","Type":"ContainerStarted","Data":"a105397454bde86398d2def21ab6576579028238dd6248edb81662fe072c67e2"}
Sep 10 14:49:55.027362 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:49:55.027313    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr" podStartSLOduration=2.027300018 podStartE2EDuration="2.027300018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:55.026390993 +0000 UTC m=+1321.452128414" watchObservedRunningTime="2026-09-10 14:49:55.027300018 +0000 UTC m=+1321.453037436"
Sep 10 14:50:03.746530 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:03.746495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:50:03.747138 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:03.746546    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:50:03.759066 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:03.759042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:50:04.059933 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:04.059862    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:50:13.435321 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.435270    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"]
Sep 10 14:50:13.440364 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.440322    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.442650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.442626    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"embedding-w0-9b1e67-kserve-self-signed-certs\""
Sep 10 14:50:13.448289 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.448267    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"]
Sep 10 14:50:13.475170 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.475142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-home\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.475289 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.475189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-dshm\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.475358 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.475282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tmp-dir\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.475419 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.475369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-model-cache\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.475419 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.475408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpkk\" (UniqueName: \"kubernetes.io/projected/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-kube-api-access-jhpkk\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.475524 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.475438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tls-certs\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.576396 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576368    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-home\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.576526 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-dshm\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.576526 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tmp-dir\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.576526 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-model-cache\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.576526 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpkk\" (UniqueName: \"kubernetes.io/projected/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-kube-api-access-jhpkk\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.576526 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tls-certs\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.576923 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-home\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.576976 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-model-cache\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.577018 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.576990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tmp-dir\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.578749 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.578722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-dshm\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.579063 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.579041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tls-certs\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.584370 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.584346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpkk\" (UniqueName: \"kubernetes.io/projected/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-kube-api-access-jhpkk\") pod \"embedding-w0-9b1e67-kserve-649f847f54-6m5hh\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") " pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:13.751983 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:13.751913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:14.023048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:14.023021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"]
Sep 10 14:50:14.024423 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:50:14.024397    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda34c977c_4c5a_4d1e_bb47_5af8e50c3edb.slice/crio-59223ca1067d8484b42b8cbf031abd72657c684ebffbf8bca7dca314cec88452 WatchSource:0}: Error finding container 59223ca1067d8484b42b8cbf031abd72657c684ebffbf8bca7dca314cec88452: Status 404 returned error can't find the container with id 59223ca1067d8484b42b8cbf031abd72657c684ebffbf8bca7dca314cec88452
Sep 10 14:50:14.099270 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:14.099238    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh" event={"ID":"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb","Type":"ContainerStarted","Data":"46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2"}
Sep 10 14:50:14.099389 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:14.099277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh" event={"ID":"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb","Type":"ContainerStarted","Data":"59223ca1067d8484b42b8cbf031abd72657c684ebffbf8bca7dca314cec88452"}
Sep 10 14:50:14.114601 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:14.114557    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh" podStartSLOduration=1.114541567 podStartE2EDuration="1.114541567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:14.113254796 +0000 UTC m=+1340.538992221" watchObservedRunningTime="2026-09-10 14:50:14.114541567 +0000 UTC m=+1340.540278986"
Sep 10 14:50:23.752831 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:23.752736    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:23.752831 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:23.752806    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:23.764927 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:23.764903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:50:24.161706 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:50:24.161667    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:51:35.033453 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.033409    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"]
Sep 10 14:51:35.036706 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.036668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.038602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.038577    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6eb765-serving-cert\""
Sep 10 14:51:35.047908 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.047887    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"]
Sep 10 14:51:35.119864 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.119839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.119981 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.119887    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-metrics-certs\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.119981 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.119916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-maas-api-tls\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.119981 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.119940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm66j\" (UniqueName: \"kubernetes.io/projected/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-kube-api-access-dm66j\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.220886 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.220854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-metrics-certs\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.221034 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.220904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-maas-api-tls\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.221034 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.220945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm66j\" (UniqueName: \"kubernetes.io/projected/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-kube-api-access-dm66j\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.221146 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.221096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.221721 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.221672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.223361 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.223339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-maas-api-tls\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.223470 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.223348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-metrics-certs\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.228122 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.228102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm66j\" (UniqueName: \"kubernetes.io/projected/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-kube-api-access-dm66j\") pod \"maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.347785 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.347722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:35.620489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.620399    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"]
Sep 10 14:51:35.622797 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:51:35.622774    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5cf5e7_1f9b_492f_b2bd_5462f1820f2b.slice/crio-22cf89af62668dce54fc56611aadf511db06cfc4a519ba5c6d6c6bf603310b6d WatchSource:0}: Error finding container 22cf89af62668dce54fc56611aadf511db06cfc4a519ba5c6d6c6bf603310b6d: Status 404 returned error can't find the container with id 22cf89af62668dce54fc56611aadf511db06cfc4a519ba5c6d6c6bf603310b6d
Sep 10 14:51:35.624109 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:35.624089    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:51:36.480509 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:36.480475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2" event={"ID":"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b","Type":"ContainerStarted","Data":"47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0"}
Sep 10 14:51:36.480961 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:36.480540    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:36.480961 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:36.480566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2" event={"ID":"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b","Type":"ContainerStarted","Data":"22cf89af62668dce54fc56611aadf511db06cfc4a519ba5c6d6c6bf603310b6d"}
Sep 10 14:51:36.496633 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:36.496588    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2" podStartSLOduration=1.006787326 podStartE2EDuration="1.49657491s" totalImagesPullingTime="489.787584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:51:36.494743654 +0000 UTC m=+1422.920481071" watchObservedRunningTime="2026-09-10 14:51:36.49657491 +0000 UTC m=+1422.922312365"
Sep 10 14:51:42.491976 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:42.491942    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:51:54.193724 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:51:54.193636    2566 scope.go:117] "RemoveContainer" containerID="783ee920cdbf5e413943309e51de57a40c2050977a49b1bf748d6247d9901f96"
Sep 10 14:52:04.147355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.147319    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"]
Sep 10 14:52:04.152200 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.152175    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.154115 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.154087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f1b717-serving-cert\""
Sep 10 14:52:04.162631 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.162572    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"]
Sep 10 14:52:04.258745 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.258714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.258880 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.258791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce599ff-e042-4a0c-9021-6c8b12329589-openshift-service-ca\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.258880 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.258834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc74f\" (UniqueName: \"kubernetes.io/projected/4ce599ff-e042-4a0c-9021-6c8b12329589-kube-api-access-jc74f\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.258990 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.258926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-metrics-certs\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.359831 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.359800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce599ff-e042-4a0c-9021-6c8b12329589-openshift-service-ca\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.359983 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.359842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc74f\" (UniqueName: \"kubernetes.io/projected/4ce599ff-e042-4a0c-9021-6c8b12329589-kube-api-access-jc74f\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.359983 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.359890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-metrics-certs\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.359983 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.359928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.360151 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:52:04.360010    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-serving-cert: secret "maas-api-e2e-api-b-f1b717-serving-cert" not found
Sep 10 14:52:04.360151 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:52:04.360061    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls podName:4ce599ff-e042-4a0c-9021-6c8b12329589 nodeName:}" failed. No retries permitted until 2026-09-10 14:52:04.860046118 +0000 UTC m=+1451.285783516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls") pod "maas-api-e2e-api-b-f1b717-78c78c785f-22qwg" (UID: "4ce599ff-e042-4a0c-9021-6c8b12329589") : secret "maas-api-e2e-api-b-f1b717-serving-cert" not found
Sep 10 14:52:04.360481 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.360458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce599ff-e042-4a0c-9021-6c8b12329589-openshift-service-ca\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.362271 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.362250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-metrics-certs\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.367612 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.367587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc74f\" (UniqueName: \"kubernetes.io/projected/4ce599ff-e042-4a0c-9021-6c8b12329589-kube-api-access-jc74f\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.865454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.865423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.867755 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.867731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls\") pod \"maas-api-e2e-api-b-f1b717-78c78c785f-22qwg\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:04.889525 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.889495    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"]
Sep 10 14:52:04.892974 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.892956    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:04.894862 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.894843    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f1b717\""
Sep 10 14:52:04.895162 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.895144    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f1b717-dockercfg-wzzlb\""
Sep 10 14:52:04.903593 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:04.903568    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"]
Sep 10 14:52:05.064141 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.064107    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:05.068174 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.068153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c133fd82-bdbb-4372-8a2c-59cff4789e85-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn\" (UID: \"c133fd82-bdbb-4372-8a2c-59cff4789e85\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:05.068244 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.068226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8fnf\" (UniqueName: \"kubernetes.io/projected/c133fd82-bdbb-4372-8a2c-59cff4789e85-kube-api-access-x8fnf\") pod \"payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn\" (UID: \"c133fd82-bdbb-4372-8a2c-59cff4789e85\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:05.168904 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.168870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8fnf\" (UniqueName: \"kubernetes.io/projected/c133fd82-bdbb-4372-8a2c-59cff4789e85-kube-api-access-x8fnf\") pod \"payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn\" (UID: \"c133fd82-bdbb-4372-8a2c-59cff4789e85\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:05.169252 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.169034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c133fd82-bdbb-4372-8a2c-59cff4789e85-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn\" (UID: \"c133fd82-bdbb-4372-8a2c-59cff4789e85\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:05.169745 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.169725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c133fd82-bdbb-4372-8a2c-59cff4789e85-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn\" (UID: \"c133fd82-bdbb-4372-8a2c-59cff4789e85\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:05.176757 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.176739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8fnf\" (UniqueName: \"kubernetes.io/projected/c133fd82-bdbb-4372-8a2c-59cff4789e85-kube-api-access-x8fnf\") pod \"payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn\" (UID: \"c133fd82-bdbb-4372-8a2c-59cff4789e85\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:05.203964 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.203945    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:05.333417 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.333392    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"]
Sep 10 14:52:05.335260 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:52:05.335235    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce599ff_e042_4a0c_9021_6c8b12329589.slice/crio-a4edae60bda49e2e698b59d70efac05ef54f18d68dbd2c2d79633602a1cc0b40 WatchSource:0}: Error finding container a4edae60bda49e2e698b59d70efac05ef54f18d68dbd2c2d79633602a1cc0b40: Status 404 returned error can't find the container with id a4edae60bda49e2e698b59d70efac05ef54f18d68dbd2c2d79633602a1cc0b40
Sep 10 14:52:05.469743 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.469721    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"]
Sep 10 14:52:05.471109 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:52:05.471079    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc133fd82_bdbb_4372_8a2c_59cff4789e85.slice/crio-3beb5e67a9e20f5ed35e27e5057dd8625e9941d73fe64763f2d9839557bbf264 WatchSource:0}: Error finding container 3beb5e67a9e20f5ed35e27e5057dd8625e9941d73fe64763f2d9839557bbf264: Status 404 returned error can't find the container with id 3beb5e67a9e20f5ed35e27e5057dd8625e9941d73fe64763f2d9839557bbf264
Sep 10 14:52:05.616493 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.616459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn" event={"ID":"c133fd82-bdbb-4372-8a2c-59cff4789e85","Type":"ContainerStarted","Data":"8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01"}
Sep 10 14:52:05.616682 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.616515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn" event={"ID":"c133fd82-bdbb-4372-8a2c-59cff4789e85","Type":"ContainerStarted","Data":"3beb5e67a9e20f5ed35e27e5057dd8625e9941d73fe64763f2d9839557bbf264"}
Sep 10 14:52:05.616682 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.616574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:05.618002 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.617978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg" event={"ID":"4ce599ff-e042-4a0c-9021-6c8b12329589","Type":"ContainerStarted","Data":"a4edae60bda49e2e698b59d70efac05ef54f18d68dbd2c2d79633602a1cc0b40"}
Sep 10 14:52:05.632401 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:05.632357    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn" podStartSLOduration=1.632344783 podStartE2EDuration="1.632344783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:52:05.631010741 +0000 UTC m=+1452.056748160" watchObservedRunningTime="2026-09-10 14:52:05.632344783 +0000 UTC m=+1452.058082236"
Sep 10 14:52:06.626392 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:06.626355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg" event={"ID":"4ce599ff-e042-4a0c-9021-6c8b12329589","Type":"ContainerStarted","Data":"be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3"}
Sep 10 14:52:06.626771 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:06.626423    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:06.643489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:06.643441    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg" podStartSLOduration=2.269015742 podStartE2EDuration="2.643426273s" totalImagesPullingTime="374.410531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:52:06.64116631 +0000 UTC m=+1453.066903728" watchObservedRunningTime="2026-09-10 14:52:06.643426273 +0000 UTC m=+1453.069163691"
Sep 10 14:52:12.637702 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:12.637653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:52:16.628813 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:16.628776    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:52:46.734586 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:46.734529    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"]
Sep 10 14:52:46.735228 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:46.734811    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2" podUID="5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" containerName="maas-api" containerID="cri-o://47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0" gracePeriod=30
Sep 10 14:52:46.986937 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:46.986872    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:52:47.153426 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.153394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-openshift-service-ca\") pod \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") "
Sep 10 14:52:47.153426 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.153426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-maas-api-tls\") pod \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") "
Sep 10 14:52:47.153629 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.153474    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-metrics-certs\") pod \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") "
Sep 10 14:52:47.153629 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.153511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm66j\" (UniqueName: \"kubernetes.io/projected/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-kube-api-access-dm66j\") pod \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\" (UID: \"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b\") "
Sep 10 14:52:47.153844 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.153815    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" (UID: "5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:52:47.155720 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.155664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" (UID: "5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:52:47.155720 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.155701    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" (UID: "5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:52:47.155876 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.155746    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-kube-api-access-dm66j" (OuterVolumeSpecName: "kube-api-access-dm66j") pod "5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" (UID: "5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b"). InnerVolumeSpecName "kube-api-access-dm66j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:52:47.255272 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.255175    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:52:47.255272 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.255207    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm66j\" (UniqueName: \"kubernetes.io/projected/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-kube-api-access-dm66j\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:52:47.255272 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.255220    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:52:47.255272 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.255233    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:52:47.829320 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.829286    2566 generic.go:356] "Generic (PLEG): container finished" podID="5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" containerID="47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0" exitCode=0
Sep 10 14:52:47.829818 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.829329    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"
Sep 10 14:52:47.829818 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.829374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2" event={"ID":"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b","Type":"ContainerDied","Data":"47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0"}
Sep 10 14:52:47.829818 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.829437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2" event={"ID":"5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b","Type":"ContainerDied","Data":"22cf89af62668dce54fc56611aadf511db06cfc4a519ba5c6d6c6bf603310b6d"}
Sep 10 14:52:47.829818 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.829478    2566 scope.go:117] "RemoveContainer" containerID="47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0"
Sep 10 14:52:47.841230 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.840486    2566 scope.go:117] "RemoveContainer" containerID="47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0"
Sep 10 14:52:47.842181 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:52:47.842141    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0\": container with ID starting with 47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0 not found: ID does not exist" containerID="47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0"
Sep 10 14:52:47.842304 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.842184    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0"} err="failed to get container status \"47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0\": rpc error: code = NotFound desc = could not find container \"47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0\": container with ID starting with 47874d7f63fcfe2ae6b370c3e7c7b7f4fa3eee6ddf6155d34232264adf2924d0 not found: ID does not exist"
Sep 10 14:52:47.851477 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.851449    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"]
Sep 10 14:52:47.860765 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:47.857392    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6eb765-846fbf47b9-h85l2"]
Sep 10 14:52:48.111438 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:48.111356    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" path="/var/lib/kubelet/pods/5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b/volumes"
Sep 10 14:52:54.085814 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:54.085777    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:52:54.088118 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:54.088094    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:52:54.089641 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:54.089623    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:52:54.092037 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:52:54.092020    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:53:09.474817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.474443    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"]
Sep 10 14:53:09.474817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.474723    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg" podUID="4ce599ff-e042-4a0c-9021-6c8b12329589" containerName="maas-api" containerID="cri-o://be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3" gracePeriod=30
Sep 10 14:53:09.528755 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.528684    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"]
Sep 10 14:53:09.529019 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.528994    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn" podUID="c133fd82-bdbb-4372-8a2c-59cff4789e85" containerName="payload-processing" containerID="cri-o://8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01" gracePeriod=30
Sep 10 14:53:09.808266 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.808244    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:53:09.811178 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.811150    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:53:09.862678 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.862645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc74f\" (UniqueName: \"kubernetes.io/projected/4ce599ff-e042-4a0c-9021-6c8b12329589-kube-api-access-jc74f\") pod \"4ce599ff-e042-4a0c-9021-6c8b12329589\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") "
Sep 10 14:53:09.862678 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.862680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c133fd82-bdbb-4372-8a2c-59cff4789e85-plugins-config-volume\") pod \"c133fd82-bdbb-4372-8a2c-59cff4789e85\" (UID: \"c133fd82-bdbb-4372-8a2c-59cff4789e85\") "
Sep 10 14:53:09.862882 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.862727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce599ff-e042-4a0c-9021-6c8b12329589-openshift-service-ca\") pod \"4ce599ff-e042-4a0c-9021-6c8b12329589\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") "
Sep 10 14:53:09.862882 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.862840    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls\") pod \"4ce599ff-e042-4a0c-9021-6c8b12329589\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") "
Sep 10 14:53:09.862882 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.862874    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-metrics-certs\") pod \"4ce599ff-e042-4a0c-9021-6c8b12329589\" (UID: \"4ce599ff-e042-4a0c-9021-6c8b12329589\") "
Sep 10 14:53:09.862977 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.862937    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8fnf\" (UniqueName: \"kubernetes.io/projected/c133fd82-bdbb-4372-8a2c-59cff4789e85-kube-api-access-x8fnf\") pod \"c133fd82-bdbb-4372-8a2c-59cff4789e85\" (UID: \"c133fd82-bdbb-4372-8a2c-59cff4789e85\") "
Sep 10 14:53:09.863058 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.863037    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce599ff-e042-4a0c-9021-6c8b12329589-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ce599ff-e042-4a0c-9021-6c8b12329589" (UID: "4ce599ff-e042-4a0c-9021-6c8b12329589"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:53:09.863355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.863328    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce599ff-e042-4a0c-9021-6c8b12329589-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:09.863474 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.863423    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c133fd82-bdbb-4372-8a2c-59cff4789e85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c133fd82-bdbb-4372-8a2c-59cff4789e85" (UID: "c133fd82-bdbb-4372-8a2c-59cff4789e85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:53:09.865013 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.864991    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce599ff-e042-4a0c-9021-6c8b12329589-kube-api-access-jc74f" (OuterVolumeSpecName: "kube-api-access-jc74f") pod "4ce599ff-e042-4a0c-9021-6c8b12329589" (UID: "4ce599ff-e042-4a0c-9021-6c8b12329589"). InnerVolumeSpecName "kube-api-access-jc74f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:53:09.865013 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.865000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ce599ff-e042-4a0c-9021-6c8b12329589" (UID: "4ce599ff-e042-4a0c-9021-6c8b12329589"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:53:09.865355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.865337    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4ce599ff-e042-4a0c-9021-6c8b12329589" (UID: "4ce599ff-e042-4a0c-9021-6c8b12329589"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:53:09.865406 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.865388    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c133fd82-bdbb-4372-8a2c-59cff4789e85-kube-api-access-x8fnf" (OuterVolumeSpecName: "kube-api-access-x8fnf") pod "c133fd82-bdbb-4372-8a2c-59cff4789e85" (UID: "c133fd82-bdbb-4372-8a2c-59cff4789e85"). InnerVolumeSpecName "kube-api-access-x8fnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:53:09.934951 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.934918    2566 generic.go:356] "Generic (PLEG): container finished" podID="c133fd82-bdbb-4372-8a2c-59cff4789e85" containerID="8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01" exitCode=0
Sep 10 14:53:09.935095 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.934966    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"
Sep 10 14:53:09.935095 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.934999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn" event={"ID":"c133fd82-bdbb-4372-8a2c-59cff4789e85","Type":"ContainerDied","Data":"8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01"}
Sep 10 14:53:09.935095 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.935053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn" event={"ID":"c133fd82-bdbb-4372-8a2c-59cff4789e85","Type":"ContainerDied","Data":"3beb5e67a9e20f5ed35e27e5057dd8625e9941d73fe64763f2d9839557bbf264"}
Sep 10 14:53:09.935095 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.935074    2566 scope.go:117] "RemoveContainer" containerID="8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01"
Sep 10 14:53:09.936426 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.936399    2566 generic.go:356] "Generic (PLEG): container finished" podID="4ce599ff-e042-4a0c-9021-6c8b12329589" containerID="be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3" exitCode=0
Sep 10 14:53:09.936539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.936443    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"
Sep 10 14:53:09.936539 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.936492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg" event={"ID":"4ce599ff-e042-4a0c-9021-6c8b12329589","Type":"ContainerDied","Data":"be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3"}
Sep 10 14:53:09.936655 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.936541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg" event={"ID":"4ce599ff-e042-4a0c-9021-6c8b12329589","Type":"ContainerDied","Data":"a4edae60bda49e2e698b59d70efac05ef54f18d68dbd2c2d79633602a1cc0b40"}
Sep 10 14:53:09.944137 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.944047    2566 scope.go:117] "RemoveContainer" containerID="8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01"
Sep 10 14:53:09.944465 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:53:09.944434    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01\": container with ID starting with 8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01 not found: ID does not exist" containerID="8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01"
Sep 10 14:53:09.944550 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.944478    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01"} err="failed to get container status \"8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01\": rpc error: code = NotFound desc = could not find container \"8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01\": container with ID starting with 8dd5da54751f696f834e1b58ab6ef250bfa900615d6241f1de5e9d8b8df69d01 not found: ID does not exist"
Sep 10 14:53:09.944550 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.944501    2566 scope.go:117] "RemoveContainer" containerID="be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3"
Sep 10 14:53:09.953602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.953586    2566 scope.go:117] "RemoveContainer" containerID="be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3"
Sep 10 14:53:09.953909 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:53:09.953889    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3\": container with ID starting with be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3 not found: ID does not exist" containerID="be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3"
Sep 10 14:53:09.953978 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.953918    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3"} err="failed to get container status \"be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3\": rpc error: code = NotFound desc = could not find container \"be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3\": container with ID starting with be2d44e0626979a84fc541462d61511da0490fd7e1ee599dbffdc667ff4aacc3 not found: ID does not exist"
Sep 10 14:53:09.964378 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.964353    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8fnf\" (UniqueName: \"kubernetes.io/projected/c133fd82-bdbb-4372-8a2c-59cff4789e85-kube-api-access-x8fnf\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:09.964470 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.964382    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc74f\" (UniqueName: \"kubernetes.io/projected/4ce599ff-e042-4a0c-9021-6c8b12329589-kube-api-access-jc74f\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:09.964470 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.964397    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c133fd82-bdbb-4372-8a2c-59cff4789e85-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:09.964470 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.964411    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:09.964470 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.964424    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ce599ff-e042-4a0c-9021-6c8b12329589-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:09.965255 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.965238    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"]
Sep 10 14:53:09.987180 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:09.987127    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b717-b5dfd4c97-knjxn"]
Sep 10 14:53:10.003143 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:10.003118    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"]
Sep 10 14:53:10.008090 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:10.008066    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1b717-78c78c785f-22qwg"]
Sep 10 14:53:10.110492 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:10.110459    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce599ff-e042-4a0c-9021-6c8b12329589" path="/var/lib/kubelet/pods/4ce599ff-e042-4a0c-9021-6c8b12329589/volumes"
Sep 10 14:53:10.110885 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:10.110870    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c133fd82-bdbb-4372-8a2c-59cff4789e85" path="/var/lib/kubelet/pods/c133fd82-bdbb-4372-8a2c-59cff4789e85/volumes"
Sep 10 14:53:27.009413 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.009379    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"]
Sep 10 14:53:27.009958 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.009628    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p" podUID="e3df5b42-c1ef-453f-91b2-4813591b49ef" containerName="maas-api" containerID="cri-o://0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454" gracePeriod=30
Sep 10 14:53:27.245188 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.245163    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:53:27.317895 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.317823    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-metrics-certs\") pod \"e3df5b42-c1ef-453f-91b2-4813591b49ef\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") "
Sep 10 14:53:27.317895 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.317867    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3df5b42-c1ef-453f-91b2-4813591b49ef-openshift-service-ca\") pod \"e3df5b42-c1ef-453f-91b2-4813591b49ef\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") "
Sep 10 14:53:27.318104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.317901    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-maas-api-tls\") pod \"e3df5b42-c1ef-453f-91b2-4813591b49ef\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") "
Sep 10 14:53:27.318104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.317963    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4kkb\" (UniqueName: \"kubernetes.io/projected/e3df5b42-c1ef-453f-91b2-4813591b49ef-kube-api-access-d4kkb\") pod \"e3df5b42-c1ef-453f-91b2-4813591b49ef\" (UID: \"e3df5b42-c1ef-453f-91b2-4813591b49ef\") "
Sep 10 14:53:27.318282 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.318249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3df5b42-c1ef-453f-91b2-4813591b49ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3df5b42-c1ef-453f-91b2-4813591b49ef" (UID: "e3df5b42-c1ef-453f-91b2-4813591b49ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:53:27.320003 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.319977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3df5b42-c1ef-453f-91b2-4813591b49ef-kube-api-access-d4kkb" (OuterVolumeSpecName: "kube-api-access-d4kkb") pod "e3df5b42-c1ef-453f-91b2-4813591b49ef" (UID: "e3df5b42-c1ef-453f-91b2-4813591b49ef"). InnerVolumeSpecName "kube-api-access-d4kkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:53:27.320355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.320329    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3df5b42-c1ef-453f-91b2-4813591b49ef" (UID: "e3df5b42-c1ef-453f-91b2-4813591b49ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:53:27.320433 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.320349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3df5b42-c1ef-453f-91b2-4813591b49ef" (UID: "e3df5b42-c1ef-453f-91b2-4813591b49ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:53:27.419682 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.419649    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kkb\" (UniqueName: \"kubernetes.io/projected/e3df5b42-c1ef-453f-91b2-4813591b49ef-kube-api-access-d4kkb\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:27.419682 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.419678    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:27.419682 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.419702    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3df5b42-c1ef-453f-91b2-4813591b49ef-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:27.419952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:27.419716    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3df5b42-c1ef-453f-91b2-4813591b49ef-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:28.035201 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.035170    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3df5b42-c1ef-453f-91b2-4813591b49ef" containerID="0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454" exitCode=0
Sep 10 14:53:28.035624 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.035218    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"
Sep 10 14:53:28.035624 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.035263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p" event={"ID":"e3df5b42-c1ef-453f-91b2-4813591b49ef","Type":"ContainerDied","Data":"0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454"}
Sep 10 14:53:28.035624 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.035313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p" event={"ID":"e3df5b42-c1ef-453f-91b2-4813591b49ef","Type":"ContainerDied","Data":"8a08b160fbf4fd0b0d1da9a05e184c9f66e2551e154080172f86ce494dcc70b1"}
Sep 10 14:53:28.035624 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.035338    2566 scope.go:117] "RemoveContainer" containerID="0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454"
Sep 10 14:53:28.044278 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.044261    2566 scope.go:117] "RemoveContainer" containerID="0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454"
Sep 10 14:53:28.044529 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:53:28.044512    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454\": container with ID starting with 0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454 not found: ID does not exist" containerID="0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454"
Sep 10 14:53:28.044585 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.044539    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454"} err="failed to get container status \"0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454\": rpc error: code = NotFound desc = could not find container \"0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454\": container with ID starting with 0d15152803bd7bc97a8ab3be4948f96387555c80237e32e2a085352e5060b454 not found: ID does not exist"
Sep 10 14:53:28.057048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.057013    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"]
Sep 10 14:53:28.061283 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.061263    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5c9641-d7fc4bd5c-c9z2p"]
Sep 10 14:53:28.110261 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:28.110235    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3df5b42-c1ef-453f-91b2-4813591b49ef" path="/var/lib/kubelet/pods/e3df5b42-c1ef-453f-91b2-4813591b49ef/volumes"
Sep 10 14:53:44.362937 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.362903    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"]
Sep 10 14:53:44.365459 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.363200    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9" podUID="4ca354d8-9d2d-4c48-a731-a44be67b3d39" containerName="payload-processing" containerID="cri-o://65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa" gracePeriod=30
Sep 10 14:53:44.613580 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.613529    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:53:44.677651 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.677623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4rc5\" (UniqueName: \"kubernetes.io/projected/4ca354d8-9d2d-4c48-a731-a44be67b3d39-kube-api-access-d4rc5\") pod \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\" (UID: \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\") "
Sep 10 14:53:44.677821 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.677664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca354d8-9d2d-4c48-a731-a44be67b3d39-plugins-config-volume\") pod \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\" (UID: \"4ca354d8-9d2d-4c48-a731-a44be67b3d39\") "
Sep 10 14:53:44.678086 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.678065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca354d8-9d2d-4c48-a731-a44be67b3d39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ca354d8-9d2d-4c48-a731-a44be67b3d39" (UID: "4ca354d8-9d2d-4c48-a731-a44be67b3d39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:53:44.679722 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.679702    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca354d8-9d2d-4c48-a731-a44be67b3d39-kube-api-access-d4rc5" (OuterVolumeSpecName: "kube-api-access-d4rc5") pod "4ca354d8-9d2d-4c48-a731-a44be67b3d39" (UID: "4ca354d8-9d2d-4c48-a731-a44be67b3d39"). InnerVolumeSpecName "kube-api-access-d4rc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:53:44.779023 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.778987    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4rc5\" (UniqueName: \"kubernetes.io/projected/4ca354d8-9d2d-4c48-a731-a44be67b3d39-kube-api-access-d4rc5\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:44.779023 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:44.779016    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca354d8-9d2d-4c48-a731-a44be67b3d39-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:53:45.115721 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.115659    2566 generic.go:356] "Generic (PLEG): container finished" podID="4ca354d8-9d2d-4c48-a731-a44be67b3d39" containerID="65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa" exitCode=0
Sep 10 14:53:45.115901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.115730    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"
Sep 10 14:53:45.115901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.115749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9" event={"ID":"4ca354d8-9d2d-4c48-a731-a44be67b3d39","Type":"ContainerDied","Data":"65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa"}
Sep 10 14:53:45.115901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.115805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9" event={"ID":"4ca354d8-9d2d-4c48-a731-a44be67b3d39","Type":"ContainerDied","Data":"b8dc5db10af902d4d55ffd4c8270c3ca63e3a99c21d71c09dcd25f09cc895f25"}
Sep 10 14:53:45.115901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.115827    2566 scope.go:117] "RemoveContainer" containerID="65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa"
Sep 10 14:53:45.124461 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.124440    2566 scope.go:117] "RemoveContainer" containerID="65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa"
Sep 10 14:53:45.124852 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:53:45.124832    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa\": container with ID starting with 65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa not found: ID does not exist" containerID="65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa"
Sep 10 14:53:45.124911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.124862    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa"} err="failed to get container status \"65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa\": rpc error: code = NotFound desc = could not find container \"65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa\": container with ID starting with 65e03147dec641d96028605021b643930412fc40d362d3c44659bb78f05644aa not found: ID does not exist"
Sep 10 14:53:45.135070 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.135041    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"]
Sep 10 14:53:45.137348 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:45.137326    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ecfa25-547c5ff96c-gbzr9"]
Sep 10 14:53:46.110093 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:53:46.110059    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca354d8-9d2d-4c48-a731-a44be67b3d39" path="/var/lib/kubelet/pods/4ca354d8-9d2d-4c48-a731-a44be67b3d39/volumes"
Sep 10 14:55:16.730873 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:16.730785    2566 controller.go:195] "Failed to update lease" err="Put \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-162.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 10 14:55:16.886298 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:16.886256    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-10T14:55:06Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-10T14:55:06Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-10T14:55:06Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-10T14:55:06Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-129-162.ec2.internal\": Patch \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-129-162.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 10 14:55:26.731621 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:26.731579    2566 controller.go:195] "Failed to update lease" err="Put \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-162.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:55:26.887304 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:26.887262    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-162.ec2.internal\": Get \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-129-162.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:55:35.847234 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:35.847196    2566 webhook.go:269] Failed to make webhook authorizer request: Post "https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews": context canceled
Sep 10 14:55:35.847234 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:35.847237    2566 server.go:347] "Authorization error" err="Post \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics"
Sep 10 14:55:36.731839 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:36.731798    2566 controller.go:195] "Failed to update lease" err="Put \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-162.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:55:36.887483 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:36.887445    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-162.ec2.internal\": Get \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-129-162.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:55:46.732254 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:46.732215    2566 controller.go:195] "Failed to update lease" err="Put \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-162.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:55:46.888538 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:46.888503    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-162.ec2.internal\": Get \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-129-162.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:55:48.236911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236867    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vw687\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.236911 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236895    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236913    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236924    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236931    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236937    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236985    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236887    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.236993    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237029    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237041    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j5sdw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237053    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237066    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237086    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237093    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-rph4b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237097    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237103    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237131    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237135    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.237421 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237156    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237210    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237197    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237237    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9b1e67\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237251    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e6941c-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237271    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dqdvj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237282    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237305    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237308    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237315    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237309    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237330    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7j4nl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237353    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237357    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qsd67\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237362    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237371    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237387    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237397    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237412    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237427    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237429    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5vzzs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237449    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237457    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237474    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fc42h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237486    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xfjkt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237492    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bzc9v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237518    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"embedding-w4-5b0323-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237525    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"unconfigured-w4-5b0323-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237581    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wn9lt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:55:48.237586    2566 controller.go:195] "Failed to update lease" err="Put \"https://ac98c1ffcb7ce44b0919176460062de8-a2d2dcf15b13ca4f.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-129-162.ec2.internal?timeout=10s\": http2: client connection lost"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237602    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237604    2566 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237624    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237648    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237672    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-12b6f6\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237706    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.238929 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237736    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237757    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"distinct-w0-9b1e67-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237795    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237812    2566 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237835    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"default-dockercfg-lrmnz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237880    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237887    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mztbj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237899    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237910    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237920    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237941    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237948    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237965    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237974    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.237985    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"embedding-w0-9b1e67-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238000    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-4qd9v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238011    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"default-dockercfg-cx7lb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238015    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.239489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238027    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238016    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238043    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238045    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6lsg4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238065    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zp2m4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238065    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238077    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238097    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"auth-test-model-12b6f6-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238100    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pxcmb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238107    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238115    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238120    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238129    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238144    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238143    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238156    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"default-dockercfg-r4d7f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238158    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238115    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240142 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238160    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lsf8q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238181    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238191    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wqb5l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238209    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238213    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238225    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238239    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238247    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-khg79\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238254    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238260    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238267    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238274    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238227    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238289    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238292    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238304    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-75k5q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238276    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-5b0323-dockercfg-st6z9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238319    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.240791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238327    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238310    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z67dn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238339    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238352    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238364    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238376    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238383    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238390    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238394    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"distinct-w4-5b0323-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238410    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238413    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238426    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238440    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-njw85\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238264    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238452    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-12b6f6-dockercfg-cr222\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238321    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bf25v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238468    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238479    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241331 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238488    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2nvw6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238494    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238452    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238508    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238513    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238516    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238366    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238523    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-5b0323\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238496    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pxrpp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238397    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238496    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238521    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pptck\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238469    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238536    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238537    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238524    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9b1e67-dockercfg-xn2p5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238563    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dh959\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238543    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.241902 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238471    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jpkgl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.242592 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238578    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:48.242592 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:48.238571    2566 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 10 14:55:49.049103 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.049067    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.058148 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.058121    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 14:55:49.058822 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.058802    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 14:55:49.071669 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.071647    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.082763 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.082739    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.084910 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.084893    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 14:55:49.099051 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.099026    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-12b6f6-dockercfg-cr222\""
Sep 10 14:55:49.099761 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.099748    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 14:55:49.111871 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.111851    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 14:55:49.114202 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.114188    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.115798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.115784    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 14:55:49.119346 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.119329    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.122561 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.122544    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 14:55:49.132356 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.132341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 14:55:49.136658 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.136642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"default-dockercfg-lrmnz\""
Sep 10 14:55:49.148132 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.148117    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.160267 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.160244    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 14:55:49.163611 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.163589    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 14:55:49.163722 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.163656    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 14:55:49.174094 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.174078    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 14:55:49.174924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.174910    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 14:55:49.175399 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.175387    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.192350 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.192327    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.192558 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.192542    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"embedding-w0-9b1e67-kserve-self-signed-certs\""
Sep 10 14:55:49.202253 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.202236    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.202953 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.202938    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wqb5l\""
Sep 10 14:55:49.203747 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.203735    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.206817 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.206802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vw687\""
Sep 10 14:55:49.216602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.216584    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.226900 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.226881    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-njw85\""
Sep 10 14:55:49.226975 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.226948    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-khg79\""
Sep 10 14:55:49.227585 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.227571    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 14:55:49.228355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.228341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.229738 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.229724    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.231264 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.231249    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.236354 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.236337    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 14:55:49.243916 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.243901    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.247422 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.247407    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jpkgl\""
Sep 10 14:55:49.248528 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.248514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:55:49.249916 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.249900    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:55:49.249987 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.249929    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:55:49.249987 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.249939    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:55:49.249987 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.249954    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-162.ec2.internal" event="NodeReady"
Sep 10 14:55:49.255215 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.255199    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 14:55:49.258983 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.258970    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.262205 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.262191    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 14:55:49.265001 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.264987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 14:55:49.269579 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.269566    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9b1e67\""
Sep 10 14:55:49.284284 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.284264    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pxrpp\""
Sep 10 14:55:49.291048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.291027    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 14:55:49.291484 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.291471    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 14:55:49.307753 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.307667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.307851 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.307840    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.311359 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.311342    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:55:49.326014 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.325993    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.336985 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.336959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 14:55:49.338593 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.338574    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 14:55:49.340437 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.340407    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.340565 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.340548    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dh959\""
Sep 10 14:55:49.344340 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.344323    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-12b6f6\"/\"auth-test-model-12b6f6-kserve-self-signed-certs\""
Sep 10 14:55:49.354716 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.354670    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.375072 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.375050    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z67dn\""
Sep 10 14:55:49.375612 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.375596    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 14:55:49.376757 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.376741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.381638 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.381622    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.383901 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.383884    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5vzzs\""
Sep 10 14:55:49.386822 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.386802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 14:55:49.396193 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.396173    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 14:55:49.396927 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.396911    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"unconfigured-w4-5b0323-kserve-self-signed-certs\""
Sep 10 14:55:49.401284 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.401264    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 14:55:49.401404 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.401386    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 14:55:49.401547 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.401532    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lsf8q\""
Sep 10 14:55:49.422495 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.422469    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.423658 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.423642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"embedding-w4-5b0323-kserve-self-signed-certs\""
Sep 10 14:55:49.428105 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.428088    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 14:55:49.434835 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.434817    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.435615 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.435600    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.447304 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.447283    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.451389 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.451370    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.463393 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.463375    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.464680 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.464655    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 14:55:49.466103 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.466089    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"distinct-w4-5b0323-kserve-self-signed-certs\""
Sep 10 14:55:49.472615 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.472598    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fc42h\""
Sep 10 14:55:49.480720 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.480675    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zp2m4\""
Sep 10 14:55:49.489588 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.489565    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 14:55:49.489678 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.489565    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"default-dockercfg-cx7lb\""
Sep 10 14:55:49.494022 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.494008    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 14:55:49.496272 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.496257    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xfjkt\""
Sep 10 14:55:49.501502 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.501486    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 14:55:49.518213 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.518194    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dqdvj\""
Sep 10 14:55:49.523240 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.523225    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 14:55:49.523333 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.523318    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 14:55:49.524278 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.524263    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.529735 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.529718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 14:55:49.537814 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.537794    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:55:49.556476 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.556459    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.557361 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.557348    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 14:55:49.575836 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.575784    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.583124 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.583105    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wn9lt\""
Sep 10 14:55:49.584860 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.584846    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.594425 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.594409    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bf25v\""
Sep 10 14:55:49.597480 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.597466    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.602000 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.601986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 14:55:49.611949 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.611936    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 14:55:49.617639 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.617625    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 14:55:49.631937 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.631918    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.632464 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.632450    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 14:55:49.632740 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.632685    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.632972 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.632960    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.637212 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.637195    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-5b0323-dockercfg-st6z9\""
Sep 10 14:55:49.644099 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.644081    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 14:55:49.653666 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.653643    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 14:55:49.655085 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.655070    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e6941c-serving-cert\""
Sep 10 14:55:49.660752 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.660736    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.661282 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.661267    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-75k5q\""
Sep 10 14:55:49.673784 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.673768    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.674067 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.674049    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.680129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.680109    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-12b6f6\""
Sep 10 14:55:49.680578 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.680564    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rph4b\""
Sep 10 14:55:49.688631 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.688613    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pxcmb\""
Sep 10 14:55:49.688815 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.688802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qsd67\""
Sep 10 14:55:49.692361 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.692339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9b1e67-dockercfg-xn2p5\""
Sep 10 14:55:49.698955 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.698936    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6lsg4\""
Sep 10 14:55:49.702163 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.702144    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.713275 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.713257    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.713378 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.713362    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 14:55:49.717184 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.717166    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pptck\""
Sep 10 14:55:49.717453 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.717437    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mztbj\""
Sep 10 14:55:49.727431 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.727412    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:55:49.728087 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.728074    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.733974 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.733959    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.744260 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.744229    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:55:49.759182 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.759162    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 14:55:49.759260 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.759231    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b0323\"/\"default-dockercfg-r4d7f\""
Sep 10 14:55:49.770266 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.770243    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 14:55:49.775232 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.775216    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j5sdw\""
Sep 10 14:55:49.775311 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.775220    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4qd9v\""
Sep 10 14:55:49.788532 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.788513    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.788662 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.788642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9b1e67\"/\"distinct-w0-9b1e67-kserve-self-signed-certs\""
Sep 10 14:55:49.804702 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.804678    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 14:55:49.804840 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.804824    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 14:55:49.811843 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.811822    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-5b0323\""
Sep 10 14:55:49.829516 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.829467    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.848976 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.848955    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 14:55:49.871893 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.871868    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 14:55:49.892514 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.892493    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2nvw6\""
Sep 10 14:55:49.908650 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.908631    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 14:55:49.931028 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.931006    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bzc9v\""
Sep 10 14:55:49.947949 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.947927    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:55:49.969571 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:49.969551    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7j4nl\""
Sep 10 14:55:57.360256 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:57.360222    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"]
Sep 10 14:55:57.373437 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:55:57.373404    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"]
Sep 10 14:56:09.291974 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.291938    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"]
Sep 10 14:56:09.292560 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.292222    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp" podUID="011df5f1-8acb-4cbd-979c-def8f6e0d1ff" containerName="payload-pre-processing" containerID="cri-o://8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336" gracePeriod=30
Sep 10 14:56:09.534479 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.534457    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:56:09.610889 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.610795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68jq9\" (UniqueName: \"kubernetes.io/projected/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-kube-api-access-68jq9\") pod \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\" (UID: \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\") "
Sep 10 14:56:09.611046 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.610913    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-plugins-config-volume\") pod \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\" (UID: \"011df5f1-8acb-4cbd-979c-def8f6e0d1ff\") "
Sep 10 14:56:09.611352 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.611320    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "011df5f1-8acb-4cbd-979c-def8f6e0d1ff" (UID: "011df5f1-8acb-4cbd-979c-def8f6e0d1ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:56:09.613104 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.613078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-kube-api-access-68jq9" (OuterVolumeSpecName: "kube-api-access-68jq9") pod "011df5f1-8acb-4cbd-979c-def8f6e0d1ff" (UID: "011df5f1-8acb-4cbd-979c-def8f6e0d1ff"). InnerVolumeSpecName "kube-api-access-68jq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:56:09.711740 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.711716    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jq9\" (UniqueName: \"kubernetes.io/projected/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-kube-api-access-68jq9\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:09.711849 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.711738    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/011df5f1-8acb-4cbd-979c-def8f6e0d1ff-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:09.798475 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.798446    2566 generic.go:356] "Generic (PLEG): container finished" podID="011df5f1-8acb-4cbd-979c-def8f6e0d1ff" containerID="8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336" exitCode=0
Sep 10 14:56:09.798615 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.798497    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"
Sep 10 14:56:09.798615 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.798528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp" event={"ID":"011df5f1-8acb-4cbd-979c-def8f6e0d1ff","Type":"ContainerDied","Data":"8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336"}
Sep 10 14:56:09.798615 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.798578    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp" event={"ID":"011df5f1-8acb-4cbd-979c-def8f6e0d1ff","Type":"ContainerDied","Data":"581ff89350f671da8cb11a0077553d2aecce756af3f7a3dc3c33b3315aaa057e"}
Sep 10 14:56:09.798615 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.798600    2566 scope.go:117] "RemoveContainer" containerID="8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336"
Sep 10 14:56:09.807345 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.807330    2566 scope.go:117] "RemoveContainer" containerID="8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336"
Sep 10 14:56:09.807583 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:09.807567    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336\": container with ID starting with 8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336 not found: ID does not exist" containerID="8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336"
Sep 10 14:56:09.807624 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.807590    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336"} err="failed to get container status \"8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336\": rpc error: code = NotFound desc = could not find container \"8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336\": container with ID starting with 8097728e909a62b62e6721aa8227964fa6812afe4e2f2b82ed257809341dd336 not found: ID does not exist"
Sep 10 14:56:09.849090 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.849064    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"]
Sep 10 14:56:09.867178 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:09.867123    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9b1e67-6d7f56f698-q5qbp"]
Sep 10 14:56:10.111565 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:10.111538    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="011df5f1-8acb-4cbd-979c-def8f6e0d1ff" path="/var/lib/kubelet/pods/011df5f1-8acb-4cbd-979c-def8f6e0d1ff/volumes"
Sep 10 14:56:12.383622 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.383585    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh" podUID="c1c50ec4-3a7b-43dd-afdb-16b000d27b02" containerName="main" containerID="cri-o://353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531" gracePeriod=45
Sep 10 14:56:12.392415 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.392378    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh" podUID="a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" containerName="main" containerID="cri-o://46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2" gracePeriod=45
Sep 10 14:56:12.647147 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.647124    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:56:12.649735 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.649715    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:56:12.739424 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tmp-dir\") pod \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") "
Sep 10 14:56:12.739424 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739430    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhpkk\" (UniqueName: \"kubernetes.io/projected/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-kube-api-access-jhpkk\") pod \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") "
Sep 10 14:56:12.739647 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739450    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tmp-dir\") pod \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") "
Sep 10 14:56:12.739647 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739490    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-home\") pod \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") "
Sep 10 14:56:12.739647 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739536    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tls-certs\") pod \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") "
Sep 10 14:56:12.739647 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739583    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-model-cache\") pod \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") "
Sep 10 14:56:12.739647 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739616    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg6zm\" (UniqueName: \"kubernetes.io/projected/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-kube-api-access-jg6zm\") pod \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") "
Sep 10 14:56:12.739946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739653    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-dshm\") pod \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") "
Sep 10 14:56:12.739946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-home\") pod \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") "
Sep 10 14:56:12.739946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739723    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tls-certs\") pod \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") "
Sep 10 14:56:12.739946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739720    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1c50ec4-3a7b-43dd-afdb-16b000d27b02" (UID: "c1c50ec4-3a7b-43dd-afdb-16b000d27b02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:12.739946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-home" (OuterVolumeSpecName: "home") pod "c1c50ec4-3a7b-43dd-afdb-16b000d27b02" (UID: "c1c50ec4-3a7b-43dd-afdb-16b000d27b02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:12.739946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-model-cache\") pod \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\" (UID: \"c1c50ec4-3a7b-43dd-afdb-16b000d27b02\") "
Sep 10 14:56:12.739946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739790    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-dshm\") pod \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\" (UID: \"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb\") "
Sep 10 14:56:12.739946 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.739803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" (UID: "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:12.740355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.740004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-home" (OuterVolumeSpecName: "home") pod "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" (UID: "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:12.740355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.740158    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.740469 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.740359    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.740469 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.740377    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.740469 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.740390    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.740469 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.740235    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-model-cache" (OuterVolumeSpecName: "model-cache") pod "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" (UID: "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:12.740469 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.740413    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1c50ec4-3a7b-43dd-afdb-16b000d27b02" (UID: "c1c50ec4-3a7b-43dd-afdb-16b000d27b02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:12.742255 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.742225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1c50ec4-3a7b-43dd-afdb-16b000d27b02" (UID: "c1c50ec4-3a7b-43dd-afdb-16b000d27b02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:56:12.742376 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.742345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-dshm" (OuterVolumeSpecName: "dshm") pod "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" (UID: "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:12.742376 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.742357    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-kube-api-access-jhpkk" (OuterVolumeSpecName: "kube-api-access-jhpkk") pod "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" (UID: "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb"). InnerVolumeSpecName "kube-api-access-jhpkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:56:12.742451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.742382    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" (UID: "a34c977c-4c5a-4d1e-bb47-5af8e50c3edb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:56:12.742451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.742407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-kube-api-access-jg6zm" (OuterVolumeSpecName: "kube-api-access-jg6zm") pod "c1c50ec4-3a7b-43dd-afdb-16b000d27b02" (UID: "c1c50ec4-3a7b-43dd-afdb-16b000d27b02"). InnerVolumeSpecName "kube-api-access-jg6zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:56:12.742538 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.742519    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-dshm" (OuterVolumeSpecName: "dshm") pod "c1c50ec4-3a7b-43dd-afdb-16b000d27b02" (UID: "c1c50ec4-3a7b-43dd-afdb-16b000d27b02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:12.814758 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.814721    2566 generic.go:356] "Generic (PLEG): container finished" podID="a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" containerID="46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2" exitCode=0
Sep 10 14:56:12.814952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.814778    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"
Sep 10 14:56:12.814952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.814795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh" event={"ID":"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb","Type":"ContainerDied","Data":"46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2"}
Sep 10 14:56:12.814952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.814838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh" event={"ID":"a34c977c-4c5a-4d1e-bb47-5af8e50c3edb","Type":"ContainerDied","Data":"59223ca1067d8484b42b8cbf031abd72657c684ebffbf8bca7dca314cec88452"}
Sep 10 14:56:12.814952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.814860    2566 scope.go:117] "RemoveContainer" containerID="46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2"
Sep 10 14:56:12.816602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.816412    2566 generic.go:356] "Generic (PLEG): container finished" podID="c1c50ec4-3a7b-43dd-afdb-16b000d27b02" containerID="353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531" exitCode=0
Sep 10 14:56:12.816602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.816524    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"
Sep 10 14:56:12.816602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.816541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh" event={"ID":"c1c50ec4-3a7b-43dd-afdb-16b000d27b02","Type":"ContainerDied","Data":"353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531"}
Sep 10 14:56:12.816602 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.816571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh" event={"ID":"c1c50ec4-3a7b-43dd-afdb-16b000d27b02","Type":"ContainerDied","Data":"9b691d220d8931ba4a04f36873de58550999ff5d39559d360d233f647e928f0e"}
Sep 10 14:56:12.826152 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.826115    2566 scope.go:117] "RemoveContainer" containerID="46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2"
Sep 10 14:56:12.826472 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:12.826450    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2\": container with ID starting with 46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2 not found: ID does not exist" containerID="46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2"
Sep 10 14:56:12.826550 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.826480    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2"} err="failed to get container status \"46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2\": rpc error: code = NotFound desc = could not find container \"46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2\": container with ID starting with 46ea655081d2dd4fddd581aee389eb21e476c764263c2c4637fd45b499532ec2 not found: ID does not exist"
Sep 10 14:56:12.826550 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.826496    2566 scope.go:117] "RemoveContainer" containerID="353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531"
Sep 10 14:56:12.836127 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.836104    2566 scope.go:117] "RemoveContainer" containerID="353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531"
Sep 10 14:56:12.836382 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:12.836364    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531\": container with ID starting with 353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531 not found: ID does not exist" containerID="353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531"
Sep 10 14:56:12.836454 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.836387    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531"} err="failed to get container status \"353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531\": rpc error: code = NotFound desc = could not find container \"353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531\": container with ID starting with 353430327a616bfe660dcffed7e02fb1b45de739420dff5773659fff128e6531 not found: ID does not exist"
Sep 10 14:56:12.840942 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.840917    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.841033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.840946    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.841033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.840961    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.841033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.840973    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpkk\" (UniqueName: \"kubernetes.io/projected/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-kube-api-access-jhpkk\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.841033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.840987    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.841033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.841001    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.841033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.841014    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg6zm\" (UniqueName: \"kubernetes.io/projected/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-kube-api-access-jg6zm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.841033 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.841026    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c50ec4-3a7b-43dd-afdb-16b000d27b02-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:12.850444 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.850412    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"]
Sep 10 14:56:12.854727 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.854686    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/embedding-w0-9b1e67-kserve-649f847f54-6m5hh"]
Sep 10 14:56:12.882951 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.882922    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"]
Sep 10 14:56:12.896436 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:12.896412    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9b1e67/distinct-w0-9b1e67-kserve-db67b855c-nq7dh"]
Sep 10 14:56:14.109743 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:14.109708    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" path="/var/lib/kubelet/pods/a34c977c-4c5a-4d1e-bb47-5af8e50c3edb/volumes"
Sep 10 14:56:14.110168 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:14.110153    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c50ec4-3a7b-43dd-afdb-16b000d27b02" path="/var/lib/kubelet/pods/c1c50ec4-3a7b-43dd-afdb-16b000d27b02/volumes"
Sep 10 14:56:27.223642 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:27.223557    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"]
Sep 10 14:56:27.239359 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:27.239330    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"]
Sep 10 14:56:27.298928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:27.298891    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"]
Sep 10 14:56:38.514988 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.514955    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"]
Sep 10 14:56:38.515385 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.515160    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x" podUID="9ade8636-3258-4ddb-90bc-1e5f91ce4093" containerName="payload-processing" containerID="cri-o://037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2" gracePeriod=30
Sep 10 14:56:38.765152 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.765095    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:56:38.859318 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.858387    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs2zb\" (UniqueName: \"kubernetes.io/projected/9ade8636-3258-4ddb-90bc-1e5f91ce4093-kube-api-access-hs2zb\") pod \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\" (UID: \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\") "
Sep 10 14:56:38.859318 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.858519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ade8636-3258-4ddb-90bc-1e5f91ce4093-plugins-config-volume\") pod \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\" (UID: \"9ade8636-3258-4ddb-90bc-1e5f91ce4093\") "
Sep 10 14:56:38.859532 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.859326    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ade8636-3258-4ddb-90bc-1e5f91ce4093-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ade8636-3258-4ddb-90bc-1e5f91ce4093" (UID: "9ade8636-3258-4ddb-90bc-1e5f91ce4093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:56:38.861843 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.861819    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ade8636-3258-4ddb-90bc-1e5f91ce4093-kube-api-access-hs2zb" (OuterVolumeSpecName: "kube-api-access-hs2zb") pod "9ade8636-3258-4ddb-90bc-1e5f91ce4093" (UID: "9ade8636-3258-4ddb-90bc-1e5f91ce4093"). InnerVolumeSpecName "kube-api-access-hs2zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:56:38.942773 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.942744    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ade8636-3258-4ddb-90bc-1e5f91ce4093" containerID="037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2" exitCode=0
Sep 10 14:56:38.942894 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.942790    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"
Sep 10 14:56:38.942894 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.942827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x" event={"ID":"9ade8636-3258-4ddb-90bc-1e5f91ce4093","Type":"ContainerDied","Data":"037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2"}
Sep 10 14:56:38.942894 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.942876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x" event={"ID":"9ade8636-3258-4ddb-90bc-1e5f91ce4093","Type":"ContainerDied","Data":"72aea228be3ad1e0f62887c213f4accff103c1b2f2d3c6643e36fab3e6f81a8a"}
Sep 10 14:56:38.942998 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.942898    2566 scope.go:117] "RemoveContainer" containerID="037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2"
Sep 10 14:56:38.957444 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.951825    2566 scope.go:117] "RemoveContainer" containerID="037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2"
Sep 10 14:56:38.957444 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:38.952182    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2\": container with ID starting with 037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2 not found: ID does not exist" containerID="037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2"
Sep 10 14:56:38.957444 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.952221    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2"} err="failed to get container status \"037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2\": rpc error: code = NotFound desc = could not find container \"037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2\": container with ID starting with 037cf6fcf3503ad09343288f9d8fd242d4eba7f38f3bf032928a315ffb205ed2 not found: ID does not exist"
Sep 10 14:56:38.959449 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.959422    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2zb\" (UniqueName: \"kubernetes.io/projected/9ade8636-3258-4ddb-90bc-1e5f91ce4093-kube-api-access-hs2zb\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:38.959449 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.959447    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ade8636-3258-4ddb-90bc-1e5f91ce4093-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:38.995719 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:38.995678    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"]
Sep 10 14:56:39.000086 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:39.000065    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5b0323-9744fd9bf-24s7x"]
Sep 10 14:56:40.109883 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:40.109851    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ade8636-3258-4ddb-90bc-1e5f91ce4093" path="/var/lib/kubelet/pods/9ade8636-3258-4ddb-90bc-1e5f91ce4093/volumes"
Sep 10 14:56:42.244451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.244412    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5" podUID="958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" containerName="main" containerID="cri-o://cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b" gracePeriod=45
Sep 10 14:56:42.258519 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.258487    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr" podUID="1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" containerName="main" containerID="cri-o://96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac" gracePeriod=45
Sep 10 14:56:42.317606 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.317570    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll" podUID="7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" containerName="main" containerID="cri-o://9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a" gracePeriod=45
Sep 10 14:56:42.593564 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.593539    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:56:42.597391 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.597372    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:56:42.600603 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.600585    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:56:42.690262 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-model-cache\") pod \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") "
Sep 10 14:56:42.690451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690283    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-dshm\") pod \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") "
Sep 10 14:56:42.690451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690321    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-dshm\") pod \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") "
Sep 10 14:56:42.690451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690348    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-home\") pod \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") "
Sep 10 14:56:42.690451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690370    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-model-cache\") pod \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") "
Sep 10 14:56:42.690451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44rsd\" (UniqueName: \"kubernetes.io/projected/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-kube-api-access-44rsd\") pod \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") "
Sep 10 14:56:42.690451 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tls-certs\") pod \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690459    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tmp-dir\") pod \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-home\") pod \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" (UID: "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-dshm\") pod \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690580    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-model-cache\") pod \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690613    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tmp-dir\") pod \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690662    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-home\") pod \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690683    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tls-certs\") pod \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\" (UID: \"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690728    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc5xr\" (UniqueName: \"kubernetes.io/projected/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-kube-api-access-mc5xr\") pod \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690751    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tmp-dir\") pod \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") "
Sep 10 14:56:42.690798 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tls-certs\") pod \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\" (UID: \"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3\") "
Sep 10 14:56:42.691408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690850    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5txwt\" (UniqueName: \"kubernetes.io/projected/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-kube-api-access-5txwt\") pod \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\" (UID: \"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3\") "
Sep 10 14:56:42.691408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.690874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" (UID: "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.691408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.691072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-home" (OuterVolumeSpecName: "home") pod "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" (UID: "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.691408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.691115    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.691408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.691131    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.691408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.691229    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" (UID: "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.691408 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.691319    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" (UID: "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.691786 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.691485    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-home" (OuterVolumeSpecName: "home") pod "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" (UID: "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.692632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.692094    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" (UID: "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.692632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.692383    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" (UID: "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.693370 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.693117    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-home" (OuterVolumeSpecName: "home") pod "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" (UID: "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.693867 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.693841    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" (UID: "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:56:42.694060 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.694031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-dshm" (OuterVolumeSpecName: "dshm") pod "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" (UID: "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.694174 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.694087    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" (UID: "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:56:42.694238 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.694214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-kube-api-access-44rsd" (OuterVolumeSpecName: "kube-api-access-44rsd") pod "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" (UID: "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d"). InnerVolumeSpecName "kube-api-access-44rsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:56:42.694294 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.694238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-dshm" (OuterVolumeSpecName: "dshm") pod "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" (UID: "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.694516 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.694489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-dshm" (OuterVolumeSpecName: "dshm") pod "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" (UID: "7fe3c3d2-3b86-4e52-91d3-a47afd61b93d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:56:42.695116 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.695092    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-kube-api-access-mc5xr" (OuterVolumeSpecName: "kube-api-access-mc5xr") pod "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" (UID: "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3"). InnerVolumeSpecName "kube-api-access-mc5xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:56:42.695339 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.695314    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" (UID: "958d61c1-0ac9-4e18-a9a2-10d2f909ecb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:56:42.695676 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.695661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-kube-api-access-5txwt" (OuterVolumeSpecName: "kube-api-access-5txwt") pod "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" (UID: "1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3"). InnerVolumeSpecName "kube-api-access-5txwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:56:42.791748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791667    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txwt\" (UniqueName: \"kubernetes.io/projected/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-kube-api-access-5txwt\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.791748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791712    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.791748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791724    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.791748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791732    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.791748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791740    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.791748 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791749    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44rsd\" (UniqueName: \"kubernetes.io/projected/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-kube-api-access-44rsd\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791756    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791765    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791772    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791779    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791786    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791793    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791799    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791807    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc5xr\" (UniqueName: \"kubernetes.io/projected/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-kube-api-access-mc5xr\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791814    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.792052 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.791821    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:42.965172 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.965137    2566 generic.go:356] "Generic (PLEG): container finished" podID="1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" containerID="96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac" exitCode=0
Sep 10 14:56:42.965332 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.965184    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"
Sep 10 14:56:42.965332 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.965223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr" event={"ID":"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3","Type":"ContainerDied","Data":"96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac"}
Sep 10 14:56:42.965332 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.965278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr" event={"ID":"1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3","Type":"ContainerDied","Data":"a105397454bde86398d2def21ab6576579028238dd6248edb81662fe072c67e2"}
Sep 10 14:56:42.965332 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.965301    2566 scope.go:117] "RemoveContainer" containerID="96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac"
Sep 10 14:56:42.966834 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.966807    2566 generic.go:356] "Generic (PLEG): container finished" podID="7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" containerID="9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a" exitCode=0
Sep 10 14:56:42.966991 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.966856    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"
Sep 10 14:56:42.966991 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.966913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll" event={"ID":"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d","Type":"ContainerDied","Data":"9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a"}
Sep 10 14:56:42.966991 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.966977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll" event={"ID":"7fe3c3d2-3b86-4e52-91d3-a47afd61b93d","Type":"ContainerDied","Data":"980c122eaa547f00e44da2cf5d529011a750d06334ee9b79a169cc0b9b0e8350"}
Sep 10 14:56:42.968295 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.968273    2566 generic.go:356] "Generic (PLEG): container finished" podID="958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" containerID="cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b" exitCode=0
Sep 10 14:56:42.968426 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.968345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5" event={"ID":"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3","Type":"ContainerDied","Data":"cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b"}
Sep 10 14:56:42.968426 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.968370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5" event={"ID":"958d61c1-0ac9-4e18-a9a2-10d2f909ecb3","Type":"ContainerDied","Data":"406e33d71944d43e7d89282eac5929019b61a5d8f0d553026f79d34f49c46e8c"}
Sep 10 14:56:42.968426 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.968347    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"
Sep 10 14:56:42.976443 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.976421    2566 scope.go:117] "RemoveContainer" containerID="96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac"
Sep 10 14:56:42.976743 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:42.976706    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac\": container with ID starting with 96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac not found: ID does not exist" containerID="96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac"
Sep 10 14:56:42.976859 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.976742    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac"} err="failed to get container status \"96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac\": rpc error: code = NotFound desc = could not find container \"96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac\": container with ID starting with 96e9eb0cc0193cf121484ef5a5f299eb191e25bf33aa820b4abe05f774d3bdac not found: ID does not exist"
Sep 10 14:56:42.976859 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.976764    2566 scope.go:117] "RemoveContainer" containerID="9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a"
Sep 10 14:56:42.984659 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.984642    2566 scope.go:117] "RemoveContainer" containerID="9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a"
Sep 10 14:56:42.984941 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:42.984923    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a\": container with ID starting with 9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a not found: ID does not exist" containerID="9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a"
Sep 10 14:56:42.985001 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.984946    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a"} err="failed to get container status \"9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a\": rpc error: code = NotFound desc = could not find container \"9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a\": container with ID starting with 9f1581f037fe9908ebe1dc45b5cc7eafbdb28a95b82671fb71cd0c8706b1390a not found: ID does not exist"
Sep 10 14:56:42.985001 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.984961    2566 scope.go:117] "RemoveContainer" containerID="cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b"
Sep 10 14:56:42.992190 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.992174    2566 scope.go:117] "RemoveContainer" containerID="cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b"
Sep 10 14:56:42.992460 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:42.992438    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b\": container with ID starting with cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b not found: ID does not exist" containerID="cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b"
Sep 10 14:56:42.992562 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:42.992462    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b"} err="failed to get container status \"cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b\": rpc error: code = NotFound desc = could not find container \"cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b\": container with ID starting with cf5aa27de002d85a58bbb2715d3e1927e9799b137934b37e0e3c52989f536e5b not found: ID does not exist"
Sep 10 14:56:43.017957 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:43.017932    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"]
Sep 10 14:56:43.034763 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:43.034738    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/embedding-w4-5b0323-kserve-6f96646559-b7zxr"]
Sep 10 14:56:43.058642 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:43.058592    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"]
Sep 10 14:56:43.080896 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:43.080872    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/distinct-w4-5b0323-kserve-7d8846bf94-4bmw5"]
Sep 10 14:56:43.111928 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:43.111906    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"]
Sep 10 14:56:43.118390 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:43.118371    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5b0323/unconfigured-w4-5b0323-kserve-b99cfc67c-2wlll"]
Sep 10 14:56:44.110629 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:44.110597    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" path="/var/lib/kubelet/pods/1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3/volumes"
Sep 10 14:56:44.111095 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:44.110969    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" path="/var/lib/kubelet/pods/7fe3c3d2-3b86-4e52-91d3-a47afd61b93d/volumes"
Sep 10 14:56:44.111291 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:44.111278    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" path="/var/lib/kubelet/pods/958d61c1-0ac9-4e18-a9a2-10d2f909ecb3/volumes"
Sep 10 14:56:50.474462 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474426    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7821-d7e86e"]
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474760    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" containerName="main"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474773    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" containerName="main"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474787    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c50ec4-3a7b-43dd-afdb-16b000d27b02" containerName="main"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474796    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c50ec4-3a7b-43dd-afdb-16b000d27b02" containerName="main"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474803    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca354d8-9d2d-4c48-a731-a44be67b3d39" containerName="payload-processing"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474809    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca354d8-9d2d-4c48-a731-a44be67b3d39" containerName="payload-processing"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474816    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce599ff-e042-4a0c-9021-6c8b12329589" containerName="maas-api"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474822    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce599ff-e042-4a0c-9021-6c8b12329589" containerName="maas-api"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474831    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011df5f1-8acb-4cbd-979c-def8f6e0d1ff" containerName="payload-pre-processing"
Sep 10 14:56:50.474833 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474835    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="011df5f1-8acb-4cbd-979c-def8f6e0d1ff" containerName="payload-pre-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474843    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474849    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474854    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3df5b42-c1ef-453f-91b2-4813591b49ef" containerName="maas-api"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474859    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3df5b42-c1ef-453f-91b2-4813591b49ef" containerName="maas-api"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474869    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ade8636-3258-4ddb-90bc-1e5f91ce4093" containerName="payload-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474877    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ade8636-3258-4ddb-90bc-1e5f91ce4093" containerName="payload-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474885    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474890    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474899    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c133fd82-bdbb-4372-8a2c-59cff4789e85" containerName="payload-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474904    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c133fd82-bdbb-4372-8a2c-59cff4789e85" containerName="payload-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474915    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" containerName="maas-api"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474920    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" containerName="maas-api"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474925    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474930    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474990    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c50ec4-3a7b-43dd-afdb-16b000d27b02" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.474997    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34c977c-4c5a-4d1e-bb47-5af8e50c3edb" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475003    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5cf5e7-1f9b-492f-b2bd-5462f1820f2b" containerName="maas-api"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475009    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ade8636-3258-4ddb-90bc-1e5f91ce4093" containerName="payload-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475014    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="011df5f1-8acb-4cbd-979c-def8f6e0d1ff" containerName="payload-pre-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475019    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="958d61c1-0ac9-4e18-a9a2-10d2f909ecb3" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475028    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca354d8-9d2d-4c48-a731-a44be67b3d39" containerName="payload-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475036    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad8c06c-0bc0-4f7a-8a5c-b0f4479393e3" containerName="main"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475042    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c133fd82-bdbb-4372-8a2c-59cff4789e85" containerName="payload-processing"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475048    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce599ff-e042-4a0c-9021-6c8b12329589" containerName="maas-api"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475053    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3df5b42-c1ef-453f-91b2-4813591b49ef" containerName="maas-api"
Sep 10 14:56:50.475129 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.475058    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe3c3d2-3b86-4e52-91d3-a47afd61b93d" containerName="main"
Sep 10 14:56:50.479461 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.479440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:50.486084 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.485906    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x6mfq\""
Sep 10 14:56:50.496348 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.496325    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7821-d7e86e"]
Sep 10 14:56:50.644610 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.644575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5sp\" (UniqueName: \"kubernetes.io/projected/8bd3fb96-78c2-40d6-ae7e-5261482983e1-kube-api-access-ch5sp\") pod \"test-curl-7821-d7e86e\" (UID: \"8bd3fb96-78c2-40d6-ae7e-5261482983e1\") " pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:50.745976 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.745899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5sp\" (UniqueName: \"kubernetes.io/projected/8bd3fb96-78c2-40d6-ae7e-5261482983e1-kube-api-access-ch5sp\") pod \"test-curl-7821-d7e86e\" (UID: \"8bd3fb96-78c2-40d6-ae7e-5261482983e1\") " pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:50.758632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.758601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5sp\" (UniqueName: \"kubernetes.io/projected/8bd3fb96-78c2-40d6-ae7e-5261482983e1-kube-api-access-ch5sp\") pod \"test-curl-7821-d7e86e\" (UID: \"8bd3fb96-78c2-40d6-ae7e-5261482983e1\") " pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:50.788632 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:50.788609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:51.071683 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:51.071659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7821-d7e86e"]
Sep 10 14:56:51.073425 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:56:51.073400    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bd3fb96_78c2_40d6_ae7e_5261482983e1.slice/crio-ee8c9a0e07f41bf20fc9565e445a12124c4ca0471d1a1e7aa8eccf036f2d7bfc WatchSource:0}: Error finding container ee8c9a0e07f41bf20fc9565e445a12124c4ca0471d1a1e7aa8eccf036f2d7bfc: Status 404 returned error can't find the container with id ee8c9a0e07f41bf20fc9565e445a12124c4ca0471d1a1e7aa8eccf036f2d7bfc
Sep 10 14:56:52.013431 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:52.013394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7821-d7e86e" event={"ID":"8bd3fb96-78c2-40d6-ae7e-5261482983e1","Type":"ContainerStarted","Data":"48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c"}
Sep 10 14:56:52.013431 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:52.013435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7821-d7e86e" event={"ID":"8bd3fb96-78c2-40d6-ae7e-5261482983e1","Type":"ContainerStarted","Data":"ee8c9a0e07f41bf20fc9565e445a12124c4ca0471d1a1e7aa8eccf036f2d7bfc"}
Sep 10 14:56:52.049627 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:52.049580    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7821-d7e86e" podStartSLOduration=2.049565655 podStartE2EDuration="2.049565655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:52.04607558 +0000 UTC m=+1738.471812998" watchObservedRunningTime="2026-09-10 14:56:52.049565655 +0000 UTC m=+1738.475303078"
Sep 10 14:56:53.147570 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:53.147537    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7821-d7e86e"]
Sep 10 14:56:53.164613 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:53.164584    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7821-d7e86e"]
Sep 10 14:56:54.022016 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:54.021975    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7821-d7e86e" podUID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" containerName="test-curl-7821-d7e86e" containerID="cri-o://48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c" gracePeriod=2
Sep 10 14:56:54.027184 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:54.027157    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7821-d7e86e\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:54.113432 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:54.113402    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7821-d7e86e\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:56.198805 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:56.198781    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:56.289627 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:56.289574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch5sp\" (UniqueName: \"kubernetes.io/projected/8bd3fb96-78c2-40d6-ae7e-5261482983e1-kube-api-access-ch5sp\") pod \"8bd3fb96-78c2-40d6-ae7e-5261482983e1\" (UID: \"8bd3fb96-78c2-40d6-ae7e-5261482983e1\") "
Sep 10 14:56:56.291459 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:56.291432    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd3fb96-78c2-40d6-ae7e-5261482983e1-kube-api-access-ch5sp" (OuterVolumeSpecName: "kube-api-access-ch5sp") pod "8bd3fb96-78c2-40d6-ae7e-5261482983e1" (UID: "8bd3fb96-78c2-40d6-ae7e-5261482983e1"). InnerVolumeSpecName "kube-api-access-ch5sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:56:56.390518 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:56.390494    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5sp\" (UniqueName: \"kubernetes.io/projected/8bd3fb96-78c2-40d6-ae7e-5261482983e1-kube-api-access-ch5sp\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:56:57.037255 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:57.037220    2566 generic.go:356] "Generic (PLEG): container finished" podID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" containerID="48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c" exitCode=137
Sep 10 14:56:57.037433 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:57.037261    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:57.037433 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:57.037295    2566 scope.go:117] "RemoveContainer" containerID="48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c"
Sep 10 14:56:57.047077 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:57.047054    2566 scope.go:117] "RemoveContainer" containerID="48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c"
Sep 10 14:56:57.047339 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:57.047322    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c\": container with ID starting with 48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c not found: ID does not exist" containerID="48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c"
Sep 10 14:56:57.047375 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:57.047347    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c"} err="failed to get container status \"48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c\": rpc error: code = NotFound desc = could not find container \"48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c\": container with ID starting with 48e10ebd74d6d5e0d2e5f6053b2546c287e3e4c371ff4d0b5b42e6cc12fe3d7c not found: ID does not exist"
Sep 10 14:56:57.050239 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:56:57.050217    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7821-d7e86e\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" pod="opendatahub/test-curl-7821-d7e86e"
Sep 10 14:56:58.111070 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:58.111031    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" path="/var/lib/kubelet/pods/8bd3fb96-78c2-40d6-ae7e-5261482983e1/volumes"
Sep 10 14:56:58.752630 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:56:58.752597    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"]
Sep 10 14:57:13.771131 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:13.771071    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj" podUID="f3ee621e-9a06-49eb-aa97-4deed322e6d0" containerName="main" containerID="cri-o://5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e" gracePeriod=45
Sep 10 14:57:14.018489 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.018466    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:57:14.025192 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tmp-dir\") pod \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") "
Sep 10 14:57:14.025192 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025165    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-home\") pod \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") "
Sep 10 14:57:14.025192 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tls-certs\") pod \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") "
Sep 10 14:57:14.025405 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025242    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-dshm\") pod \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") "
Sep 10 14:57:14.025405 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025277    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-model-cache\") pod \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") "
Sep 10 14:57:14.025405 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025329    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfgmr\" (UniqueName: \"kubernetes.io/projected/f3ee621e-9a06-49eb-aa97-4deed322e6d0-kube-api-access-kfgmr\") pod \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\" (UID: \"f3ee621e-9a06-49eb-aa97-4deed322e6d0\") "
Sep 10 14:57:14.025405 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025364    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3ee621e-9a06-49eb-aa97-4deed322e6d0" (UID: "f3ee621e-9a06-49eb-aa97-4deed322e6d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:14.025405 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025384    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-home" (OuterVolumeSpecName: "home") pod "f3ee621e-9a06-49eb-aa97-4deed322e6d0" (UID: "f3ee621e-9a06-49eb-aa97-4deed322e6d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:14.025669 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025540    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3ee621e-9a06-49eb-aa97-4deed322e6d0" (UID: "f3ee621e-9a06-49eb-aa97-4deed322e6d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:14.025669 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025600    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:57:14.025669 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025617    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:57:14.025669 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.025632    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:57:14.027391 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.027365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-dshm" (OuterVolumeSpecName: "dshm") pod "f3ee621e-9a06-49eb-aa97-4deed322e6d0" (UID: "f3ee621e-9a06-49eb-aa97-4deed322e6d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:14.027479 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.027395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3ee621e-9a06-49eb-aa97-4deed322e6d0" (UID: "f3ee621e-9a06-49eb-aa97-4deed322e6d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:57:14.027479 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.027405    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ee621e-9a06-49eb-aa97-4deed322e6d0-kube-api-access-kfgmr" (OuterVolumeSpecName: "kube-api-access-kfgmr") pod "f3ee621e-9a06-49eb-aa97-4deed322e6d0" (UID: "f3ee621e-9a06-49eb-aa97-4deed322e6d0"). InnerVolumeSpecName "kube-api-access-kfgmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:57:14.112917 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.112884    2566 generic.go:356] "Generic (PLEG): container finished" podID="f3ee621e-9a06-49eb-aa97-4deed322e6d0" containerID="5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e" exitCode=0
Sep 10 14:57:14.113048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.112922    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"
Sep 10 14:57:14.113048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.112969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj" event={"ID":"f3ee621e-9a06-49eb-aa97-4deed322e6d0","Type":"ContainerDied","Data":"5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e"}
Sep 10 14:57:14.113048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.113009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj" event={"ID":"f3ee621e-9a06-49eb-aa97-4deed322e6d0","Type":"ContainerDied","Data":"5b3cf551755f3c8325af386553688c9ade12cc8d6070c4c10f059f31ac02a572"}
Sep 10 14:57:14.113048 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.113029    2566 scope.go:117] "RemoveContainer" containerID="5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e"
Sep 10 14:57:14.121452 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.121437    2566 scope.go:117] "RemoveContainer" containerID="5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e"
Sep 10 14:57:14.121814 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:57:14.121788    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e\": container with ID starting with 5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e not found: ID does not exist" containerID="5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e"
Sep 10 14:57:14.121924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.121826    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e"} err="failed to get container status \"5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e\": rpc error: code = NotFound desc = could not find container \"5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e\": container with ID starting with 5964b624abc9ca6f7b5d6d59a8b3381d08837a79ad3a45227acec2f433ef4a1e not found: ID does not exist"
Sep 10 14:57:14.126390 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.126374    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ee621e-9a06-49eb-aa97-4deed322e6d0-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:57:14.126448 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.126392    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ee621e-9a06-49eb-aa97-4deed322e6d0-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:57:14.126448 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.126401    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgmr\" (UniqueName: \"kubernetes.io/projected/f3ee621e-9a06-49eb-aa97-4deed322e6d0-kube-api-access-kfgmr\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:57:14.141612 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.141593    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"]
Sep 10 14:57:14.151529 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:14.151507    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12b6f6/auth-test-model-12b6f6-kserve-85d89cd87f-pdxqj"]
Sep 10 14:57:16.110138 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:16.110104    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ee621e-9a06-49eb-aa97-4deed322e6d0" path="/var/lib/kubelet/pods/f3ee621e-9a06-49eb-aa97-4deed322e6d0/volumes"
Sep 10 14:57:54.111932 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:54.111900    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:57:54.115032 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:54.115006    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 14:57:54.115700 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:54.115666    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:57:54.118986 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:57:54.118955    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 14:58:33.711791 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.711761    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7821-6c11fa"]
Sep 10 14:58:33.712246 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.712083    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ee621e-9a06-49eb-aa97-4deed322e6d0" containerName="main"
Sep 10 14:58:33.712246 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.712096    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ee621e-9a06-49eb-aa97-4deed322e6d0" containerName="main"
Sep 10 14:58:33.712246 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.712106    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" containerName="test-curl-7821-d7e86e"
Sep 10 14:58:33.712246 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.712111    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" containerName="test-curl-7821-d7e86e"
Sep 10 14:58:33.712246 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.712159    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ee621e-9a06-49eb-aa97-4deed322e6d0" containerName="main"
Sep 10 14:58:33.712246 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.712169    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd3fb96-78c2-40d6-ae7e-5261482983e1" containerName="test-curl-7821-d7e86e"
Sep 10 14:58:33.714912 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.714897    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-6c11fa"
Sep 10 14:58:33.717827 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.717808    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x6mfq\""
Sep 10 14:58:33.727418 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.727395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7821-6c11fa"]
Sep 10 14:58:33.741784 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.741755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqsmp\" (UniqueName: \"kubernetes.io/projected/b4f8e252-415b-4ffd-84e2-267663e83971-kube-api-access-tqsmp\") pod \"test-curl-7821-6c11fa\" (UID: \"b4f8e252-415b-4ffd-84e2-267663e83971\") " pod="opendatahub/test-curl-7821-6c11fa"
Sep 10 14:58:33.842801 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.842764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqsmp\" (UniqueName: \"kubernetes.io/projected/b4f8e252-415b-4ffd-84e2-267663e83971-kube-api-access-tqsmp\") pod \"test-curl-7821-6c11fa\" (UID: \"b4f8e252-415b-4ffd-84e2-267663e83971\") " pod="opendatahub/test-curl-7821-6c11fa"
Sep 10 14:58:33.859346 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:33.859322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqsmp\" (UniqueName: \"kubernetes.io/projected/b4f8e252-415b-4ffd-84e2-267663e83971-kube-api-access-tqsmp\") pod \"test-curl-7821-6c11fa\" (UID: \"b4f8e252-415b-4ffd-84e2-267663e83971\") " pod="opendatahub/test-curl-7821-6c11fa"
Sep 10 14:58:34.023631 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:34.023546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-6c11fa"
Sep 10 14:58:34.298278 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:34.298206    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7821-6c11fa"]
Sep 10 14:58:34.301282 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:58:34.301253    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4f8e252_415b_4ffd_84e2_267663e83971.slice/crio-47a9c7373a37b9ac9ca3a408e2023f0d24efb323d5043dc5dcf239692e213ce3 WatchSource:0}: Error finding container 47a9c7373a37b9ac9ca3a408e2023f0d24efb323d5043dc5dcf239692e213ce3: Status 404 returned error can't find the container with id 47a9c7373a37b9ac9ca3a408e2023f0d24efb323d5043dc5dcf239692e213ce3
Sep 10 14:58:34.453766 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:34.453735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7821-6c11fa" event={"ID":"b4f8e252-415b-4ffd-84e2-267663e83971","Type":"ContainerStarted","Data":"2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a"}
Sep 10 14:58:34.453766 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:34.453772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7821-6c11fa" event={"ID":"b4f8e252-415b-4ffd-84e2-267663e83971","Type":"ContainerStarted","Data":"47a9c7373a37b9ac9ca3a408e2023f0d24efb323d5043dc5dcf239692e213ce3"}
Sep 10 14:58:35.165536 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.165440    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7821-6c11fa" podStartSLOduration=2.165424717 podStartE2EDuration="2.165424717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:34.479660872 +0000 UTC m=+1840.905398304" watchObservedRunningTime="2026-09-10 14:58:35.165424717 +0000 UTC m=+1841.591162137"
Sep 10 14:58:35.166848 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.166827    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7821-6c11fa"]
Sep 10 14:58:35.173658 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.173635    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7821-6c11fa"]
Sep 10 14:58:35.311513 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.311486    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7821-5bba68"]
Sep 10 14:58:35.311793 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.311780    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f8e252-415b-4ffd-84e2-267663e83971" containerName="test-curl-7821-6c11fa"
Sep 10 14:58:35.311846 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.311795    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f8e252-415b-4ffd-84e2-267663e83971" containerName="test-curl-7821-6c11fa"
Sep 10 14:58:35.311883 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.311860    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f8e252-415b-4ffd-84e2-267663e83971" containerName="test-curl-7821-6c11fa"
Sep 10 14:58:35.314718 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.314681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-5bba68"
Sep 10 14:58:35.322813 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.322787    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7821-5bba68"]
Sep 10 14:58:35.353396 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.353365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pgg\" (UniqueName: \"kubernetes.io/projected/d9b83f5e-322f-4ea4-a897-de423ba895d0-kube-api-access-k9pgg\") pod \"test-curl-7821-5bba68\" (UID: \"d9b83f5e-322f-4ea4-a897-de423ba895d0\") " pod="opendatahub/test-curl-7821-5bba68"
Sep 10 14:58:35.454119 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.454045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pgg\" (UniqueName: \"kubernetes.io/projected/d9b83f5e-322f-4ea4-a897-de423ba895d0-kube-api-access-k9pgg\") pod \"test-curl-7821-5bba68\" (UID: \"d9b83f5e-322f-4ea4-a897-de423ba895d0\") " pod="opendatahub/test-curl-7821-5bba68"
Sep 10 14:58:35.458175 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:58:35.458156    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7821-6c11fa" container="test-curl-7821-6c11fa"
Sep 10 14:58:35.465299 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.465277    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pgg\" (UniqueName: \"kubernetes.io/projected/d9b83f5e-322f-4ea4-a897-de423ba895d0-kube-api-access-k9pgg\") pod \"test-curl-7821-5bba68\" (UID: \"d9b83f5e-322f-4ea4-a897-de423ba895d0\") " pod="opendatahub/test-curl-7821-5bba68"
Sep 10 14:58:35.624592 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.624563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-5bba68"
Sep 10 14:58:35.905964 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:35.905940    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7821-5bba68"]
Sep 10 14:58:35.908430 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:58:35.908399    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9b83f5e_322f_4ea4_a897_de423ba895d0.slice/crio-e78ee0f11464878507a0753570a2dc90b86f09ac39f57efcfb10dfc33bcd2601 WatchSource:0}: Error finding container e78ee0f11464878507a0753570a2dc90b86f09ac39f57efcfb10dfc33bcd2601: Status 404 returned error can't find the container with id e78ee0f11464878507a0753570a2dc90b86f09ac39f57efcfb10dfc33bcd2601
Sep 10 14:58:36.464407 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:36.464357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7821-5bba68" event={"ID":"d9b83f5e-322f-4ea4-a897-de423ba895d0","Type":"ContainerStarted","Data":"65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7"}
Sep 10 14:58:36.464924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:36.464422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7821-5bba68" event={"ID":"d9b83f5e-322f-4ea4-a897-de423ba895d0","Type":"ContainerStarted","Data":"e78ee0f11464878507a0753570a2dc90b86f09ac39f57efcfb10dfc33bcd2601"}
Sep 10 14:58:36.464924 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:36.464516    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7821-6c11fa" podUID="b4f8e252-415b-4ffd-84e2-267663e83971" containerName="test-curl-7821-6c11fa" containerID="cri-o://2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a" gracePeriod=2
Sep 10 14:58:36.488268 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:36.488213    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7821-5bba68" podStartSLOduration=1.488198034 podStartE2EDuration="1.488198034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:36.48553156 +0000 UTC m=+1842.911268979" watchObservedRunningTime="2026-09-10 14:58:36.488198034 +0000 UTC m=+1842.913935452"
Sep 10 14:58:37.180375 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:37.180300    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7821-5bba68"]
Sep 10 14:58:37.190655 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:37.190629    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7821-5bba68"]
Sep 10 14:58:37.469405 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:58:37.469334    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7821-5bba68" container="test-curl-7821-5bba68"
Sep 10 14:58:38.474459 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:38.474414    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7821-5bba68" podUID="d9b83f5e-322f-4ea4-a897-de423ba895d0" containerName="test-curl-7821-5bba68" containerID="cri-o://65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7" gracePeriod=2
Sep 10 14:58:38.638006 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:38.637986    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-6c11fa"
Sep 10 14:58:38.680207 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:38.680181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqsmp\" (UniqueName: \"kubernetes.io/projected/b4f8e252-415b-4ffd-84e2-267663e83971-kube-api-access-tqsmp\") pod \"b4f8e252-415b-4ffd-84e2-267663e83971\" (UID: \"b4f8e252-415b-4ffd-84e2-267663e83971\") "
Sep 10 14:58:38.682066 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:38.682041    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f8e252-415b-4ffd-84e2-267663e83971-kube-api-access-tqsmp" (OuterVolumeSpecName: "kube-api-access-tqsmp") pod "b4f8e252-415b-4ffd-84e2-267663e83971" (UID: "b4f8e252-415b-4ffd-84e2-267663e83971"). InnerVolumeSpecName "kube-api-access-tqsmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:58:38.781089 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:38.781037    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqsmp\" (UniqueName: \"kubernetes.io/projected/b4f8e252-415b-4ffd-84e2-267663e83971-kube-api-access-tqsmp\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:58:39.479732 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:39.479676    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4f8e252-415b-4ffd-84e2-267663e83971" containerID="2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a" exitCode=137
Sep 10 14:58:39.480225 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:39.479736    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-6c11fa"
Sep 10 14:58:39.480225 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:39.479735    2566 scope.go:117] "RemoveContainer" containerID="2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a"
Sep 10 14:58:39.487918 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:39.487901    2566 scope.go:117] "RemoveContainer" containerID="2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a"
Sep 10 14:58:39.488199 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:58:39.488167    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a\": container with ID starting with 2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a not found: ID does not exist" containerID="2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a"
Sep 10 14:58:39.488266 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:39.488211    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a"} err="failed to get container status \"2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a\": rpc error: code = NotFound desc = could not find container \"2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a\": container with ID starting with 2bc4c014e08b7da2955a56bb602e672eb5abd2bd342b8cc083e7072c7956534a not found: ID does not exist"
Sep 10 14:58:39.489749 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:58:39.489729    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7821-6c11fa\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="b4f8e252-415b-4ffd-84e2-267663e83971" pod="opendatahub/test-curl-7821-6c11fa"
Sep 10 14:58:40.110261 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:40.110224    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f8e252-415b-4ffd-84e2-267663e83971" path="/var/lib/kubelet/pods/b4f8e252-415b-4ffd-84e2-267663e83971/volumes"
Sep 10 14:58:40.645804 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:40.645779    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-5bba68"
Sep 10 14:58:40.696914 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:40.696890    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9pgg\" (UniqueName: \"kubernetes.io/projected/d9b83f5e-322f-4ea4-a897-de423ba895d0-kube-api-access-k9pgg\") pod \"d9b83f5e-322f-4ea4-a897-de423ba895d0\" (UID: \"d9b83f5e-322f-4ea4-a897-de423ba895d0\") "
Sep 10 14:58:40.698717 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:40.698677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b83f5e-322f-4ea4-a897-de423ba895d0-kube-api-access-k9pgg" (OuterVolumeSpecName: "kube-api-access-k9pgg") pod "d9b83f5e-322f-4ea4-a897-de423ba895d0" (UID: "d9b83f5e-322f-4ea4-a897-de423ba895d0"). InnerVolumeSpecName "kube-api-access-k9pgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:58:40.798368 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:40.798319    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9pgg\" (UniqueName: \"kubernetes.io/projected/d9b83f5e-322f-4ea4-a897-de423ba895d0-kube-api-access-k9pgg\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 14:58:41.491065 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:41.491033    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9b83f5e-322f-4ea4-a897-de423ba895d0" containerID="65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7" exitCode=137
Sep 10 14:58:41.491277 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:41.491077    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-5bba68"
Sep 10 14:58:41.491277 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:41.491133    2566 scope.go:117] "RemoveContainer" containerID="65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7"
Sep 10 14:58:41.499672 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:41.499654    2566 scope.go:117] "RemoveContainer" containerID="65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7"
Sep 10 14:58:41.499979 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:58:41.499957    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7\": container with ID starting with 65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7 not found: ID does not exist" containerID="65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7"
Sep 10 14:58:41.500040 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:41.499986    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7"} err="failed to get container status \"65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7\": rpc error: code = NotFound desc = could not find container \"65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7\": container with ID starting with 65155b83c782edd18faacaa480ddb8a2d9e619ed24d1b86d56002de5f1e0ceb7 not found: ID does not exist"
Sep 10 14:58:41.501356 ip-10-0-129-162 kubenswrapper[2566]: E0910 14:58:41.501336    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7821-5bba68\" is forbidden: User \"system:node:ip-10-0-129-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-162.ec2.internal' and this object" podUID="d9b83f5e-322f-4ea4-a897-de423ba895d0" pod="opendatahub/test-curl-7821-5bba68"
Sep 10 14:58:42.110328 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:58:42.110296    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9b83f5e-322f-4ea4-a897-de423ba895d0" path="/var/lib/kubelet/pods/d9b83f5e-322f-4ea4-a897-de423ba895d0/volumes"
Sep 10 14:59:25.705950 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.705865    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"]
Sep 10 14:59:25.706415 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.706200    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9b83f5e-322f-4ea4-a897-de423ba895d0" containerName="test-curl-7821-5bba68"
Sep 10 14:59:25.706415 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.706215    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9b83f5e-322f-4ea4-a897-de423ba895d0" containerName="test-curl-7821-5bba68"
Sep 10 14:59:25.706415 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.706269    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9b83f5e-322f-4ea4-a897-de423ba895d0" containerName="test-curl-7821-5bba68"
Sep 10 14:59:25.708898 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.708879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.717035 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.717009    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e6941c\"/\"rate-test-model-e6941c-kserve-self-signed-certs\""
Sep 10 14:59:25.717146 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.717072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e6941c\"/\"default-dockercfg-xkqhb\""
Sep 10 14:59:25.720781 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.720687    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e6941c\"/\"openshift-service-ca.crt\""
Sep 10 14:59:25.720930 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.720913    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e6941c\"/\"kube-root-ca.crt\""
Sep 10 14:59:25.745923 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.745887    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"]
Sep 10 14:59:25.844848 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.844820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-dshm\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.844952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.844857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j5hx\" (UniqueName: \"kubernetes.io/projected/5456b638-affa-4f64-828e-b403f62c2c18-kube-api-access-6j5hx\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.844952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.844905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-tmp-dir\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.844952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.844927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-home\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.844952 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.844943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5456b638-affa-4f64-828e-b403f62c2c18-tls-certs\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.845097 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.844980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-model-cache\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.945947 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.945916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-dshm\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.946063 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.945953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j5hx\" (UniqueName: \"kubernetes.io/projected/5456b638-affa-4f64-828e-b403f62c2c18-kube-api-access-6j5hx\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.946063 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.946000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-tmp-dir\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.946063 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.946028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-home\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.946063 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.946054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5456b638-affa-4f64-828e-b403f62c2c18-tls-certs\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.946266 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.946084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-model-cache\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.946452 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.946423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-home\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.946550 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.946445    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-tmp-dir\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.946550 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.946461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-model-cache\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.948068 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.948044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-dshm\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.948308 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.948289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5456b638-affa-4f64-828e-b403f62c2c18-tls-certs\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:25.985729 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:25.985659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j5hx\" (UniqueName: \"kubernetes.io/projected/5456b638-affa-4f64-828e-b403f62c2c18-kube-api-access-6j5hx\") pod \"rate-test-model-e6941c-kserve-68f7645f95-bshhc\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") " pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:26.019050 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:26.019017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:26.308800 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:26.308775    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"]
Sep 10 14:59:26.310520 ip-10-0-129-162 kubenswrapper[2566]: W0910 14:59:26.310493    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5456b638_affa_4f64_828e_b403f62c2c18.slice/crio-fcd204f130dbbc6b11a4f9303c1127932cdf4cb11d876fab9a00f9216d23f161 WatchSource:0}: Error finding container fcd204f130dbbc6b11a4f9303c1127932cdf4cb11d876fab9a00f9216d23f161: Status 404 returned error can't find the container with id fcd204f130dbbc6b11a4f9303c1127932cdf4cb11d876fab9a00f9216d23f161
Sep 10 14:59:26.687552 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:26.687521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc" event={"ID":"5456b638-affa-4f64-828e-b403f62c2c18","Type":"ContainerStarted","Data":"a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835"}
Sep 10 14:59:26.687745 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:26.687562    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc" event={"ID":"5456b638-affa-4f64-828e-b403f62c2c18","Type":"ContainerStarted","Data":"fcd204f130dbbc6b11a4f9303c1127932cdf4cb11d876fab9a00f9216d23f161"}
Sep 10 14:59:26.717134 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:26.717076    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc" podStartSLOduration=1.7170586829999999 podStartE2EDuration="1.717058683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:26.715895421 +0000 UTC m=+1893.141632841" watchObservedRunningTime="2026-09-10 14:59:26.717058683 +0000 UTC m=+1893.142796104"
Sep 10 14:59:36.019355 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:36.019311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:36.019923 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:36.019410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:36.032425 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:36.032398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 14:59:36.742884 ip-10-0-129-162 kubenswrapper[2566]: I0910 14:59:36.742855    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 15:00:00.146163 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.146127    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"]
Sep 10 15:00:00.149253 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.149230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"
Sep 10 15:00:00.151927 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.151905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gd94c\""
Sep 10 15:00:00.161074 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.161056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"
Sep 10 15:00:00.182609 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.182539    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"]
Sep 10 15:00:00.451727 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.451669    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"]
Sep 10 15:00:00.454288 ip-10-0-129-162 kubenswrapper[2566]: W0910 15:00:00.454261    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7739ee30_af7f_4a83_8450_38b99d3faf26.slice/crio-befca1dd6714f22f8ae22e46b23ef9a7f8f9b88ec168c524db4e9c4c80d51224 WatchSource:0}: Error finding container befca1dd6714f22f8ae22e46b23ef9a7f8f9b88ec168c524db4e9c4c80d51224: Status 404 returned error can't find the container with id befca1dd6714f22f8ae22e46b23ef9a7f8f9b88ec168c524db4e9c4c80d51224
Sep 10 15:00:00.843726 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.843668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf" event={"ID":"7739ee30-af7f-4a83-8450-38b99d3faf26","Type":"ContainerStarted","Data":"c87c13fc14b30fa0cc891dacb85389f7fba8a99bcf55398c10b738fb7ea22f13"}
Sep 10 15:00:00.843928 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.843776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf" event={"ID":"7739ee30-af7f-4a83-8450-38b99d3faf26","Type":"ContainerStarted","Data":"befca1dd6714f22f8ae22e46b23ef9a7f8f9b88ec168c524db4e9c4c80d51224"}
Sep 10 15:00:00.867004 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:00.866955    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf" podStartSLOduration=0.866941612 podStartE2EDuration="866.941612ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:00.862941073 +0000 UTC m=+1927.288678492" watchObservedRunningTime="2026-09-10 15:00:00.866941612 +0000 UTC m=+1927.292679030"
Sep 10 15:00:01.849402 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:01.849358    2566 generic.go:356] "Generic (PLEG): container finished" podID="7739ee30-af7f-4a83-8450-38b99d3faf26" containerID="c87c13fc14b30fa0cc891dacb85389f7fba8a99bcf55398c10b738fb7ea22f13" exitCode=0
Sep 10 15:00:01.849887 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:01.849415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf" event={"ID":"7739ee30-af7f-4a83-8450-38b99d3faf26","Type":"ContainerDied","Data":"c87c13fc14b30fa0cc891dacb85389f7fba8a99bcf55398c10b738fb7ea22f13"}
Sep 10 15:00:02.976719 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:02.976670    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"
Sep 10 15:00:03.859122 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:03.859095    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"
Sep 10 15:00:03.859297 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:03.859096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf" event={"ID":"7739ee30-af7f-4a83-8450-38b99d3faf26","Type":"ContainerDied","Data":"befca1dd6714f22f8ae22e46b23ef9a7f8f9b88ec168c524db4e9c4c80d51224"}
Sep 10 15:00:03.859297 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:03.859217    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="befca1dd6714f22f8ae22e46b23ef9a7f8f9b88ec168c524db4e9c4c80d51224"
Sep 10 15:00:11.324273 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:11.324236    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"]
Sep 10 15:00:17.671920 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.671887    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"]
Sep 10 15:00:17.672302 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.672193    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7739ee30-af7f-4a83-8450-38b99d3faf26" containerName="cleanup"
Sep 10 15:00:17.672302 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.672204    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7739ee30-af7f-4a83-8450-38b99d3faf26" containerName="cleanup"
Sep 10 15:00:17.672302 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.672270    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7739ee30-af7f-4a83-8450-38b99d3faf26" containerName="cleanup"
Sep 10 15:00:17.678254 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.678226    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.702426 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.702403    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ae0f24-serving-cert\""
Sep 10 15:00:17.736281 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.736259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.736382 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.736285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.736382 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.736329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqwm2\" (UniqueName: \"kubernetes.io/projected/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-kube-api-access-lqwm2\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.736454 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.736386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.771851 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.771815    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"]
Sep 10 15:00:17.836970 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.836941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.837098 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.836976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.837098 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.836995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.837098 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:00:17.837087    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-serving-cert: secret "maas-api-e2e-ipp-a-ae0f24-serving-cert" not found
Sep 10 15:00:17.837220 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.837117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqwm2\" (UniqueName: \"kubernetes.io/projected/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-kube-api-access-lqwm2\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.837220 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:00:17.837138    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls podName:c05ce102-65da-4ceb-ab97-eaa1aff76a3e nodeName:}" failed. No retries permitted until 2026-09-10 15:00:18.337122755 +0000 UTC m=+1944.762860152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls") pod "maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6" (UID: "c05ce102-65da-4ceb-ab97-eaa1aff76a3e") : secret "maas-api-e2e-ipp-a-ae0f24-serving-cert" not found
Sep 10 15:00:17.837658 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.837637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.839408 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.839387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:17.868461 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:17.868436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqwm2\" (UniqueName: \"kubernetes.io/projected/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-kube-api-access-lqwm2\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:18.341307 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:18.341262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:18.343538 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:18.343518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:18.588334 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:18.588301    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:18.890981 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:18.890907    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"]
Sep 10 15:00:18.894363 ip-10-0-129-162 kubenswrapper[2566]: W0910 15:00:18.894333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05ce102_65da_4ceb_ab97_eaa1aff76a3e.slice/crio-192215d45c980503cdff76ddccd9ad5e851c4107c6dcfecd3f98a0fd20600c5d WatchSource:0}: Error finding container 192215d45c980503cdff76ddccd9ad5e851c4107c6dcfecd3f98a0fd20600c5d: Status 404 returned error can't find the container with id 192215d45c980503cdff76ddccd9ad5e851c4107c6dcfecd3f98a0fd20600c5d
Sep 10 15:00:18.896054 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:18.896036    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:00:18.925188 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:18.925165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6" event={"ID":"c05ce102-65da-4ceb-ab97-eaa1aff76a3e","Type":"ContainerStarted","Data":"192215d45c980503cdff76ddccd9ad5e851c4107c6dcfecd3f98a0fd20600c5d"}
Sep 10 15:00:20.936168 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:20.936133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6" event={"ID":"c05ce102-65da-4ceb-ab97-eaa1aff76a3e","Type":"ContainerStarted","Data":"1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395"}
Sep 10 15:00:20.936559 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:20.936269    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:20.966646 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:20.966591    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6" podStartSLOduration=2.412512081 podStartE2EDuration="3.96657038s" totalImagesPullingTime="1.554058299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:20.965312771 +0000 UTC m=+1947.391050201" watchObservedRunningTime="2026-09-10 15:00:20.96657038 +0000 UTC m=+1947.392307799"
Sep 10 15:00:26.342992 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.342930    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc" podUID="5456b638-affa-4f64-828e-b403f62c2c18" containerName="main" containerID="cri-o://a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835" gracePeriod=45
Sep 10 15:00:26.583576 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.583550    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 15:00:26.711399 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711373    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-dshm\") pod \"5456b638-affa-4f64-828e-b403f62c2c18\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") "
Sep 10 15:00:26.711567 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j5hx\" (UniqueName: \"kubernetes.io/projected/5456b638-affa-4f64-828e-b403f62c2c18-kube-api-access-6j5hx\") pod \"5456b638-affa-4f64-828e-b403f62c2c18\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") "
Sep 10 15:00:26.711567 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711423    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5456b638-affa-4f64-828e-b403f62c2c18-tls-certs\") pod \"5456b638-affa-4f64-828e-b403f62c2c18\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") "
Sep 10 15:00:26.711567 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711459    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-home\") pod \"5456b638-affa-4f64-828e-b403f62c2c18\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") "
Sep 10 15:00:26.711567 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-model-cache\") pod \"5456b638-affa-4f64-828e-b403f62c2c18\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") "
Sep 10 15:00:26.711567 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711532    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-tmp-dir\") pod \"5456b638-affa-4f64-828e-b403f62c2c18\" (UID: \"5456b638-affa-4f64-828e-b403f62c2c18\") "
Sep 10 15:00:26.711886 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711772    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-home" (OuterVolumeSpecName: "home") pod "5456b638-affa-4f64-828e-b403f62c2c18" (UID: "5456b638-affa-4f64-828e-b403f62c2c18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:26.711886 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711795    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-model-cache" (OuterVolumeSpecName: "model-cache") pod "5456b638-affa-4f64-828e-b403f62c2c18" (UID: "5456b638-affa-4f64-828e-b403f62c2c18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:26.711886 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.711863    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5456b638-affa-4f64-828e-b403f62c2c18" (UID: "5456b638-affa-4f64-828e-b403f62c2c18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:26.713632 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.713611    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-dshm" (OuterVolumeSpecName: "dshm") pod "5456b638-affa-4f64-828e-b403f62c2c18" (UID: "5456b638-affa-4f64-828e-b403f62c2c18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:00:26.713726 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.713702    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5456b638-affa-4f64-828e-b403f62c2c18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5456b638-affa-4f64-828e-b403f62c2c18" (UID: "5456b638-affa-4f64-828e-b403f62c2c18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:00:26.713726 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.713715    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5456b638-affa-4f64-828e-b403f62c2c18-kube-api-access-6j5hx" (OuterVolumeSpecName: "kube-api-access-6j5hx") pod "5456b638-affa-4f64-828e-b403f62c2c18" (UID: "5456b638-affa-4f64-828e-b403f62c2c18"). InnerVolumeSpecName "kube-api-access-6j5hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:00:26.812224 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.812197    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-model-cache\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:00:26.812328 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.812241    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-tmp-dir\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:00:26.812328 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.812250    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-dshm\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:00:26.812328 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.812259    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j5hx\" (UniqueName: \"kubernetes.io/projected/5456b638-affa-4f64-828e-b403f62c2c18-kube-api-access-6j5hx\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:00:26.812328 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.812268    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5456b638-affa-4f64-828e-b403f62c2c18-tls-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:00:26.812328 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.812276    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5456b638-affa-4f64-828e-b403f62c2c18-home\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:00:26.946199 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.946173    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:00:26.965873 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.965813    2566 generic.go:356] "Generic (PLEG): container finished" podID="5456b638-affa-4f64-828e-b403f62c2c18" containerID="a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835" exitCode=0
Sep 10 15:00:26.965873 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.965848    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"
Sep 10 15:00:26.966047 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.965903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc" event={"ID":"5456b638-affa-4f64-828e-b403f62c2c18","Type":"ContainerDied","Data":"a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835"}
Sep 10 15:00:26.966047 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.965956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc" event={"ID":"5456b638-affa-4f64-828e-b403f62c2c18","Type":"ContainerDied","Data":"fcd204f130dbbc6b11a4f9303c1127932cdf4cb11d876fab9a00f9216d23f161"}
Sep 10 15:00:26.966047 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.965978    2566 scope.go:117] "RemoveContainer" containerID="a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835"
Sep 10 15:00:26.974822 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.974801    2566 scope.go:117] "RemoveContainer" containerID="a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835"
Sep 10 15:00:26.975096 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:00:26.975072    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835\": container with ID starting with a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835 not found: ID does not exist" containerID="a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835"
Sep 10 15:00:26.975153 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.975105    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835"} err="failed to get container status \"a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835\": rpc error: code = NotFound desc = could not find container \"a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835\": container with ID starting with a0e6c037686d3d4005f70d7860e3a2342a8251d23ddb4206e7eae5833ee65835 not found: ID does not exist"
Sep 10 15:00:26.999748 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:26.999721    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"]
Sep 10 15:00:27.009194 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:27.009159    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e6941c/rate-test-model-e6941c-kserve-68f7645f95-bshhc"]
Sep 10 15:00:28.110055 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:28.110019    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5456b638-affa-4f64-828e-b403f62c2c18" path="/var/lib/kubelet/pods/5456b638-affa-4f64-828e-b403f62c2c18/volumes"
Sep 10 15:00:52.128095 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.128015    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"]
Sep 10 15:00:52.128511 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.128316    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5456b638-affa-4f64-828e-b403f62c2c18" containerName="main"
Sep 10 15:00:52.128511 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.128328    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5456b638-affa-4f64-828e-b403f62c2c18" containerName="main"
Sep 10 15:00:52.128511 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.128385    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5456b638-affa-4f64-828e-b403f62c2c18" containerName="main"
Sep 10 15:00:52.132363 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.132346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.135403 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.135384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b8f4d6-serving-cert\""
Sep 10 15:00:52.207333 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.207302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce4160-4842-4a5f-be43-09e75d504ea3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.207506 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.207344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vq8\" (UniqueName: \"kubernetes.io/projected/5fce4160-4842-4a5f-be43-09e75d504ea3-kube-api-access-c2vq8\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.207506 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.207372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-metrics-certs\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.207506 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.207439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.236207 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.236167    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"]
Sep 10 15:00:52.308305 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.308269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce4160-4842-4a5f-be43-09e75d504ea3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.308305 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.308310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vq8\" (UniqueName: \"kubernetes.io/projected/5fce4160-4842-4a5f-be43-09e75d504ea3-kube-api-access-c2vq8\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.308525 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.308331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-metrics-certs\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.308525 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.308366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.308525 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:00:52.308511    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-serving-cert: secret "maas-api-e2e-ipp-b-b8f4d6-serving-cert" not found
Sep 10 15:00:52.308645 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:00:52.308585    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls podName:5fce4160-4842-4a5f-be43-09e75d504ea3 nodeName:}" failed. No retries permitted until 2026-09-10 15:00:52.808564961 +0000 UTC m=+1979.234302374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls") pod "maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q" (UID: "5fce4160-4842-4a5f-be43-09e75d504ea3") : secret "maas-api-e2e-ipp-b-b8f4d6-serving-cert" not found
Sep 10 15:00:52.309071 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.309036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce4160-4842-4a5f-be43-09e75d504ea3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.310881 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.310861    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-metrics-certs\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.439648 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.439615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vq8\" (UniqueName: \"kubernetes.io/projected/5fce4160-4842-4a5f-be43-09e75d504ea3-kube-api-access-c2vq8\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.812116 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.812025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:52.814345 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:52.814325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:53.042156 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:53.042113    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:53.349116 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:53.349035    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"]
Sep 10 15:00:53.354905 ip-10-0-129-162 kubenswrapper[2566]: W0910 15:00:53.354865    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fce4160_4842_4a5f_be43_09e75d504ea3.slice/crio-718457d7cffebde909213406e7b3b07e98890b3c17fc799990eb267d15d038b5 WatchSource:0}: Error finding container 718457d7cffebde909213406e7b3b07e98890b3c17fc799990eb267d15d038b5: Status 404 returned error can't find the container with id 718457d7cffebde909213406e7b3b07e98890b3c17fc799990eb267d15d038b5
Sep 10 15:00:54.085918 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.085887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q" event={"ID":"5fce4160-4842-4a5f-be43-09e75d504ea3","Type":"ContainerStarted","Data":"718457d7cffebde909213406e7b3b07e98890b3c17fc799990eb267d15d038b5"}
Sep 10 15:00:54.368551 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.368472    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"]
Sep 10 15:00:54.372841 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.372823    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:54.379101 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.379075    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b8f4d6-dockercfg-vjlnq\""
Sep 10 15:00:54.379204 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.379075    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b8f4d6\""
Sep 10 15:00:54.416672 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.416640    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"]
Sep 10 15:00:54.423948 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.423919    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nsml\" (UniqueName: \"kubernetes.io/projected/a3c71753-9692-454b-8477-9a0d17eb7a64-kube-api-access-9nsml\") pod \"payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f\" (UID: \"a3c71753-9692-454b-8477-9a0d17eb7a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:54.424090 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.423972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c71753-9692-454b-8477-9a0d17eb7a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f\" (UID: \"a3c71753-9692-454b-8477-9a0d17eb7a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:54.524906 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.524867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nsml\" (UniqueName: \"kubernetes.io/projected/a3c71753-9692-454b-8477-9a0d17eb7a64-kube-api-access-9nsml\") pod \"payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f\" (UID: \"a3c71753-9692-454b-8477-9a0d17eb7a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:54.525086 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.524923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c71753-9692-454b-8477-9a0d17eb7a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f\" (UID: \"a3c71753-9692-454b-8477-9a0d17eb7a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:54.525670 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.525650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c71753-9692-454b-8477-9a0d17eb7a64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f\" (UID: \"a3c71753-9692-454b-8477-9a0d17eb7a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:54.573566 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.573529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nsml\" (UniqueName: \"kubernetes.io/projected/a3c71753-9692-454b-8477-9a0d17eb7a64-kube-api-access-9nsml\") pod \"payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f\" (UID: \"a3c71753-9692-454b-8477-9a0d17eb7a64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:54.682159 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.682128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:54.971818 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:54.971681    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"]
Sep 10 15:00:54.974298 ip-10-0-129-162 kubenswrapper[2566]: W0910 15:00:54.974273    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c71753_9692_454b_8477_9a0d17eb7a64.slice/crio-15dbea100f0ed9bca80c399e6ddb1d66328c0fe3e67de2cc20a58229c8f59d5c WatchSource:0}: Error finding container 15dbea100f0ed9bca80c399e6ddb1d66328c0fe3e67de2cc20a58229c8f59d5c: Status 404 returned error can't find the container with id 15dbea100f0ed9bca80c399e6ddb1d66328c0fe3e67de2cc20a58229c8f59d5c
Sep 10 15:00:55.091616 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:55.091580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f" event={"ID":"a3c71753-9692-454b-8477-9a0d17eb7a64","Type":"ContainerStarted","Data":"f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1"}
Sep 10 15:00:55.091798 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:55.091646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f" event={"ID":"a3c71753-9692-454b-8477-9a0d17eb7a64","Type":"ContainerStarted","Data":"15dbea100f0ed9bca80c399e6ddb1d66328c0fe3e67de2cc20a58229c8f59d5c"}
Sep 10 15:00:55.091798 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:55.091680    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:00:55.093181 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:55.093156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q" event={"ID":"5fce4160-4842-4a5f-be43-09e75d504ea3","Type":"ContainerStarted","Data":"0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec"}
Sep 10 15:00:55.093293 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:55.093257    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:00:55.198297 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:55.198245    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f" podStartSLOduration=1.198226384 podStartE2EDuration="1.198226384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:55.146508115 +0000 UTC m=+1981.572245536" watchObservedRunningTime="2026-09-10 15:00:55.198226384 +0000 UTC m=+1981.623963802"
Sep 10 15:00:55.198710 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:00:55.198669    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q" podStartSLOduration=2.449142551 podStartE2EDuration="3.198661977s" totalImagesPullingTime="749.519426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:55.195021821 +0000 UTC m=+1981.620759239" watchObservedRunningTime="2026-09-10 15:00:55.198661977 +0000 UTC m=+1981.624399407"
Sep 10 15:01:01.103885 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:01:01.103853    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:01:06.100074 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:01:06.100045    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:01:20.954763 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:01:20.954729    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f_a3c71753-9692-454b-8477-9a0d17eb7a64/payload-processing/0.log"
Sep 10 15:02:07.345609 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:07.345573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f_a3c71753-9692-454b-8477-9a0d17eb7a64/payload-processing/0.log"
Sep 10 15:02:12.916627 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:12.916587    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"]
Sep 10 15:02:12.919774 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:12.919750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:12.922418 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:12.922386    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-fd1096-dockercfg-9876p\""
Sep 10 15:02:12.922501 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:12.922415    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-fd1096\""
Sep 10 15:02:12.933901 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:12.933878    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"]
Sep 10 15:02:13.085702 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:13.085668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd\" (UID: \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:13.085863 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:13.085732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ffk\" (UniqueName: \"kubernetes.io/projected/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-kube-api-access-s4ffk\") pod \"payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd\" (UID: \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:13.186991 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:13.186929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd\" (UID: \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:13.186991 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:13.186959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ffk\" (UniqueName: \"kubernetes.io/projected/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-kube-api-access-s4ffk\") pod \"payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd\" (UID: \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:13.187614 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:13.187596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd\" (UID: \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:13.197044 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:13.197022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ffk\" (UniqueName: \"kubernetes.io/projected/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-kube-api-access-s4ffk\") pod \"payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd\" (UID: \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:13.229860 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:13.229836    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:13.500586 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:13.500562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"]
Sep 10 15:02:13.502723 ip-10-0-129-162 kubenswrapper[2566]: W0910 15:02:13.502676    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1252c0f0_f065_4f4c_a98a_2de4ecfa531b.slice/crio-bbff4bff2b2404bd0cc6892e7a6ad7c1bb852f6195b3f15768fa4936346f2cc3 WatchSource:0}: Error finding container bbff4bff2b2404bd0cc6892e7a6ad7c1bb852f6195b3f15768fa4936346f2cc3: Status 404 returned error can't find the container with id bbff4bff2b2404bd0cc6892e7a6ad7c1bb852f6195b3f15768fa4936346f2cc3
Sep 10 15:02:14.439535 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:14.439502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd" event={"ID":"1252c0f0-f065-4f4c-a98a-2de4ecfa531b","Type":"ContainerStarted","Data":"21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde"}
Sep 10 15:02:14.439937 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:14.439550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd" event={"ID":"1252c0f0-f065-4f4c-a98a-2de4ecfa531b","Type":"ContainerStarted","Data":"bbff4bff2b2404bd0cc6892e7a6ad7c1bb852f6195b3f15768fa4936346f2cc3"}
Sep 10 15:02:14.439937 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:14.439607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:14.455747 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:14.455685    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd" podStartSLOduration=2.455672912 podStartE2EDuration="2.455672912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:14.455347446 +0000 UTC m=+2060.881084891" watchObservedRunningTime="2026-09-10 15:02:14.455672912 +0000 UTC m=+2060.881410330"
Sep 10 15:02:25.446598 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:25.446503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:43.267738 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.267687    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"]
Sep 10 15:02:43.268302 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.267922    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd" podUID="1252c0f0-f065-4f4c-a98a-2de4ecfa531b" containerName="payload-processing" containerID="cri-o://21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde" gracePeriod=30
Sep 10 15:02:43.515289 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.515263    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:43.574871 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.574775    2566 generic.go:356] "Generic (PLEG): container finished" podID="1252c0f0-f065-4f4c-a98a-2de4ecfa531b" containerID="21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde" exitCode=0
Sep 10 15:02:43.574871 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.574818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd" event={"ID":"1252c0f0-f065-4f4c-a98a-2de4ecfa531b","Type":"ContainerDied","Data":"21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde"}
Sep 10 15:02:43.574871 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.574863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd" event={"ID":"1252c0f0-f065-4f4c-a98a-2de4ecfa531b","Type":"ContainerDied","Data":"bbff4bff2b2404bd0cc6892e7a6ad7c1bb852f6195b3f15768fa4936346f2cc3"}
Sep 10 15:02:43.575171 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.574886    2566 scope.go:117] "RemoveContainer" containerID="21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde"
Sep 10 15:02:43.575171 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.574825    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"
Sep 10 15:02:43.583028 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.583009    2566 scope.go:117] "RemoveContainer" containerID="21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde"
Sep 10 15:02:43.583308 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:02:43.583281    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde\": container with ID starting with 21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde not found: ID does not exist" containerID="21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde"
Sep 10 15:02:43.583379 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.583318    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde"} err="failed to get container status \"21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde\": rpc error: code = NotFound desc = could not find container \"21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde\": container with ID starting with 21b84d0620474ce145613d262baa5d3808b2dc6623fa16e49f1da2bf8c1f0fde not found: ID does not exist"
Sep 10 15:02:43.607815 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.607785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-plugins-config-volume\") pod \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\" (UID: \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\") "
Sep 10 15:02:43.607968 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.607852    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4ffk\" (UniqueName: \"kubernetes.io/projected/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-kube-api-access-s4ffk\") pod \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\" (UID: \"1252c0f0-f065-4f4c-a98a-2de4ecfa531b\") "
Sep 10 15:02:43.608175 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.608154    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1252c0f0-f065-4f4c-a98a-2de4ecfa531b" (UID: "1252c0f0-f065-4f4c-a98a-2de4ecfa531b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:02:43.609930 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.609907    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-kube-api-access-s4ffk" (OuterVolumeSpecName: "kube-api-access-s4ffk") pod "1252c0f0-f065-4f4c-a98a-2de4ecfa531b" (UID: "1252c0f0-f065-4f4c-a98a-2de4ecfa531b"). InnerVolumeSpecName "kube-api-access-s4ffk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:02:43.708593 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.708562    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:02:43.708593 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.708590    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ffk\" (UniqueName: \"kubernetes.io/projected/1252c0f0-f065-4f4c-a98a-2de4ecfa531b-kube-api-access-s4ffk\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:02:43.899870 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.899833    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"]
Sep 10 15:02:43.907907 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:43.907882    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fd1096-57c8c7c9dc-4xmzd"]
Sep 10 15:02:44.110298 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:44.110264    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1252c0f0-f065-4f4c-a98a-2de4ecfa531b" path="/var/lib/kubelet/pods/1252c0f0-f065-4f4c-a98a-2de4ecfa531b/volumes"
Sep 10 15:02:54.137132 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:54.137099    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 15:02:54.140858 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:54.140835    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 15:02:54.141253 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:54.141232    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 15:02:54.145054 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:02:54.145035    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 15:03:10.072374 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.072337    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"]
Sep 10 15:03:10.072946 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.072592    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6" podUID="c05ce102-65da-4ceb-ab97-eaa1aff76a3e" containerName="maas-api" containerID="cri-o://1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395" gracePeriod=30
Sep 10 15:03:10.323108 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.323038    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:03:10.426675 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.426634    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls\") pod \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") "
Sep 10 15:03:10.426890 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.426686    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-openshift-service-ca\") pod \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") "
Sep 10 15:03:10.426890 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.426748    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-metrics-certs\") pod \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") "
Sep 10 15:03:10.427018 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.426908    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqwm2\" (UniqueName: \"kubernetes.io/projected/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-kube-api-access-lqwm2\") pod \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\" (UID: \"c05ce102-65da-4ceb-ab97-eaa1aff76a3e\") "
Sep 10 15:03:10.427196 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.427159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c05ce102-65da-4ceb-ab97-eaa1aff76a3e" (UID: "c05ce102-65da-4ceb-ab97-eaa1aff76a3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:03:10.428987 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.428958    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c05ce102-65da-4ceb-ab97-eaa1aff76a3e" (UID: "c05ce102-65da-4ceb-ab97-eaa1aff76a3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:03:10.428987 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.428959    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-kube-api-access-lqwm2" (OuterVolumeSpecName: "kube-api-access-lqwm2") pod "c05ce102-65da-4ceb-ab97-eaa1aff76a3e" (UID: "c05ce102-65da-4ceb-ab97-eaa1aff76a3e"). InnerVolumeSpecName "kube-api-access-lqwm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:03:10.429123 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.429009    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c05ce102-65da-4ceb-ab97-eaa1aff76a3e" (UID: "c05ce102-65da-4ceb-ab97-eaa1aff76a3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:03:10.527862 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.527822    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:03:10.527862 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.527856    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:03:10.527862 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.527870    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:03:10.528093 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.527882    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqwm2\" (UniqueName: \"kubernetes.io/projected/c05ce102-65da-4ceb-ab97-eaa1aff76a3e-kube-api-access-lqwm2\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:03:10.691955 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.691921    2566 generic.go:356] "Generic (PLEG): container finished" podID="c05ce102-65da-4ceb-ab97-eaa1aff76a3e" containerID="1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395" exitCode=0
Sep 10 15:03:10.692130 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.691967    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"
Sep 10 15:03:10.692130 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.692004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6" event={"ID":"c05ce102-65da-4ceb-ab97-eaa1aff76a3e","Type":"ContainerDied","Data":"1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395"}
Sep 10 15:03:10.692130 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.692053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6" event={"ID":"c05ce102-65da-4ceb-ab97-eaa1aff76a3e","Type":"ContainerDied","Data":"192215d45c980503cdff76ddccd9ad5e851c4107c6dcfecd3f98a0fd20600c5d"}
Sep 10 15:03:10.692130 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.692074    2566 scope.go:117] "RemoveContainer" containerID="1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395"
Sep 10 15:03:10.701279 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.701258    2566 scope.go:117] "RemoveContainer" containerID="1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395"
Sep 10 15:03:10.701552 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:03:10.701533    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395\": container with ID starting with 1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395 not found: ID does not exist" containerID="1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395"
Sep 10 15:03:10.701600 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.701560    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395"} err="failed to get container status \"1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395\": rpc error: code = NotFound desc = could not find container \"1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395\": container with ID starting with 1a1ffa5f113a32ed9ae0f0ae71ec7bbb6097331bf4e1562a135ae312c99a0395 not found: ID does not exist"
Sep 10 15:03:10.712206 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.712180    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"]
Sep 10 15:03:10.717955 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:10.717920    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae0f24-65b8c6c58f-wrdp6"]
Sep 10 15:03:12.110277 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:03:12.110243    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c05ce102-65da-4ceb-ab97-eaa1aff76a3e" path="/var/lib/kubelet/pods/c05ce102-65da-4ceb-ab97-eaa1aff76a3e/volumes"
Sep 10 15:04:12.952618 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:12.952585    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"]
Sep 10 15:04:12.953201 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:12.952863    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q" podUID="5fce4160-4842-4a5f-be43-09e75d504ea3" containerName="maas-api" containerID="cri-o://0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec" gracePeriod=30
Sep 10 15:04:13.086477 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.086446    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"]
Sep 10 15:04:13.086770 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.086727    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f" podUID="a3c71753-9692-454b-8477-9a0d17eb7a64" containerName="payload-processing" containerID="cri-o://f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1" gracePeriod=30
Sep 10 15:04:13.209294 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.209267    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:04:13.306340 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.306316    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls\") pod \"5fce4160-4842-4a5f-be43-09e75d504ea3\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") "
Sep 10 15:04:13.306511 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.306383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2vq8\" (UniqueName: \"kubernetes.io/projected/5fce4160-4842-4a5f-be43-09e75d504ea3-kube-api-access-c2vq8\") pod \"5fce4160-4842-4a5f-be43-09e75d504ea3\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") "
Sep 10 15:04:13.306511 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.306430    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce4160-4842-4a5f-be43-09e75d504ea3-openshift-service-ca\") pod \"5fce4160-4842-4a5f-be43-09e75d504ea3\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") "
Sep 10 15:04:13.306511 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.306471    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-metrics-certs\") pod \"5fce4160-4842-4a5f-be43-09e75d504ea3\" (UID: \"5fce4160-4842-4a5f-be43-09e75d504ea3\") "
Sep 10 15:04:13.306823 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.306763    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fce4160-4842-4a5f-be43-09e75d504ea3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fce4160-4842-4a5f-be43-09e75d504ea3" (UID: "5fce4160-4842-4a5f-be43-09e75d504ea3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:04:13.307017 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.306906    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce4160-4842-4a5f-be43-09e75d504ea3-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:13.308491 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.308465    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5fce4160-4842-4a5f-be43-09e75d504ea3" (UID: "5fce4160-4842-4a5f-be43-09e75d504ea3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:04:13.308590 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.308518    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fce4160-4842-4a5f-be43-09e75d504ea3" (UID: "5fce4160-4842-4a5f-be43-09e75d504ea3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:04:13.308877 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.308860    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fce4160-4842-4a5f-be43-09e75d504ea3-kube-api-access-c2vq8" (OuterVolumeSpecName: "kube-api-access-c2vq8") pod "5fce4160-4842-4a5f-be43-09e75d504ea3" (UID: "5fce4160-4842-4a5f-be43-09e75d504ea3"). InnerVolumeSpecName "kube-api-access-c2vq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:13.323562 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.323544    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:04:13.407362 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.407333    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c71753-9692-454b-8477-9a0d17eb7a64-plugins-config-volume\") pod \"a3c71753-9692-454b-8477-9a0d17eb7a64\" (UID: \"a3c71753-9692-454b-8477-9a0d17eb7a64\") "
Sep 10 15:04:13.407497 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.407403    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nsml\" (UniqueName: \"kubernetes.io/projected/a3c71753-9692-454b-8477-9a0d17eb7a64-kube-api-access-9nsml\") pod \"a3c71753-9692-454b-8477-9a0d17eb7a64\" (UID: \"a3c71753-9692-454b-8477-9a0d17eb7a64\") "
Sep 10 15:04:13.407836 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.407683    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:13.407836 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.407720    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2vq8\" (UniqueName: \"kubernetes.io/projected/5fce4160-4842-4a5f-be43-09e75d504ea3-kube-api-access-c2vq8\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:13.407836 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.407734    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce4160-4842-4a5f-be43-09e75d504ea3-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:13.407836 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.407737    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c71753-9692-454b-8477-9a0d17eb7a64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3c71753-9692-454b-8477-9a0d17eb7a64" (UID: "a3c71753-9692-454b-8477-9a0d17eb7a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:04:13.409937 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.409913    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c71753-9692-454b-8477-9a0d17eb7a64-kube-api-access-9nsml" (OuterVolumeSpecName: "kube-api-access-9nsml") pod "a3c71753-9692-454b-8477-9a0d17eb7a64" (UID: "a3c71753-9692-454b-8477-9a0d17eb7a64"). InnerVolumeSpecName "kube-api-access-9nsml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:13.509121 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.509050    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nsml\" (UniqueName: \"kubernetes.io/projected/a3c71753-9692-454b-8477-9a0d17eb7a64-kube-api-access-9nsml\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:13.509121 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.509074    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c71753-9692-454b-8477-9a0d17eb7a64-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:13.980876 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.980850    2566 generic.go:356] "Generic (PLEG): container finished" podID="5fce4160-4842-4a5f-be43-09e75d504ea3" containerID="0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec" exitCode=0
Sep 10 15:04:13.981216 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.980895    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"
Sep 10 15:04:13.981216 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.980938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q" event={"ID":"5fce4160-4842-4a5f-be43-09e75d504ea3","Type":"ContainerDied","Data":"0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec"}
Sep 10 15:04:13.981216 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.980993    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q" event={"ID":"5fce4160-4842-4a5f-be43-09e75d504ea3","Type":"ContainerDied","Data":"718457d7cffebde909213406e7b3b07e98890b3c17fc799990eb267d15d038b5"}
Sep 10 15:04:13.981216 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.981022    2566 scope.go:117] "RemoveContainer" containerID="0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec"
Sep 10 15:04:13.982396 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.982380    2566 generic.go:356] "Generic (PLEG): container finished" podID="a3c71753-9692-454b-8477-9a0d17eb7a64" containerID="f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1" exitCode=0
Sep 10 15:04:13.982464 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.982423    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"
Sep 10 15:04:13.982464 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.982447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f" event={"ID":"a3c71753-9692-454b-8477-9a0d17eb7a64","Type":"ContainerDied","Data":"f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1"}
Sep 10 15:04:13.982563 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.982481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f" event={"ID":"a3c71753-9692-454b-8477-9a0d17eb7a64","Type":"ContainerDied","Data":"15dbea100f0ed9bca80c399e6ddb1d66328c0fe3e67de2cc20a58229c8f59d5c"}
Sep 10 15:04:13.990429 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.990405    2566 scope.go:117] "RemoveContainer" containerID="0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec"
Sep 10 15:04:13.990675 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:04:13.990652    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec\": container with ID starting with 0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec not found: ID does not exist" containerID="0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec"
Sep 10 15:04:13.990807 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.990684    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec"} err="failed to get container status \"0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec\": rpc error: code = NotFound desc = could not find container \"0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec\": container with ID starting with 0fff5c1157c90691ad4ec4f5d76992a9124a8218c78b9454fd80413fd72816ec not found: ID does not exist"
Sep 10 15:04:13.990807 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.990728    2566 scope.go:117] "RemoveContainer" containerID="f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1"
Sep 10 15:04:13.998013 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.997997    2566 scope.go:117] "RemoveContainer" containerID="f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1"
Sep 10 15:04:13.998281 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:04:13.998266    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1\": container with ID starting with f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1 not found: ID does not exist" containerID="f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1"
Sep 10 15:04:13.998332 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:13.998285    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1"} err="failed to get container status \"f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1\": rpc error: code = NotFound desc = could not find container \"f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1\": container with ID starting with f461e373df4e7a5764be130e17e2264ededbad736e7562c0358579b2f07b34b1 not found: ID does not exist"
Sep 10 15:04:14.015218 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:14.015200    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"]
Sep 10 15:04:14.019134 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:14.019114    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b8f4d6-db77484ff-9kw4f"]
Sep 10 15:04:14.033391 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:14.033370    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"]
Sep 10 15:04:14.036585 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:14.036566    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b8f4d6-7cd79d8f5b-p5l2q"]
Sep 10 15:04:14.110005 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:14.109982    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fce4160-4842-4a5f-be43-09e75d504ea3" path="/var/lib/kubelet/pods/5fce4160-4842-4a5f-be43-09e75d504ea3/volumes"
Sep 10 15:04:14.110395 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:14.110376    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3c71753-9692-454b-8477-9a0d17eb7a64" path="/var/lib/kubelet/pods/a3c71753-9692-454b-8477-9a0d17eb7a64/volumes"
Sep 10 15:04:32.905476 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:32.905440    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"]
Sep 10 15:04:32.926773 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:32.926740    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12b6f6-29817540-2gmvf"]
Sep 10 15:04:33.023048 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.023012    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"]
Sep 10 15:04:33.023330 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.023301    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg" podUID="e3e574fc-35a9-4acc-84f3-6b437bb03925" containerName="payload-pre-processing" containerID="cri-o://a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744" gracePeriod=30
Sep 10 15:04:33.268304 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.268277    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 15:04:33.347615 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.347585    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz7fb\" (UniqueName: \"kubernetes.io/projected/e3e574fc-35a9-4acc-84f3-6b437bb03925-kube-api-access-mz7fb\") pod \"e3e574fc-35a9-4acc-84f3-6b437bb03925\" (UID: \"e3e574fc-35a9-4acc-84f3-6b437bb03925\") "
Sep 10 15:04:33.347830 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.347664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e574fc-35a9-4acc-84f3-6b437bb03925-plugins-config-volume\") pod \"e3e574fc-35a9-4acc-84f3-6b437bb03925\" (UID: \"e3e574fc-35a9-4acc-84f3-6b437bb03925\") "
Sep 10 15:04:33.348069 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.348041    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e574fc-35a9-4acc-84f3-6b437bb03925-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3e574fc-35a9-4acc-84f3-6b437bb03925" (UID: "e3e574fc-35a9-4acc-84f3-6b437bb03925"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:04:33.349703 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.349671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e574fc-35a9-4acc-84f3-6b437bb03925-kube-api-access-mz7fb" (OuterVolumeSpecName: "kube-api-access-mz7fb") pod "e3e574fc-35a9-4acc-84f3-6b437bb03925" (UID: "e3e574fc-35a9-4acc-84f3-6b437bb03925"). InnerVolumeSpecName "kube-api-access-mz7fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:33.448795 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.448670    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz7fb\" (UniqueName: \"kubernetes.io/projected/e3e574fc-35a9-4acc-84f3-6b437bb03925-kube-api-access-mz7fb\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:33.448795 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:33.448726    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e574fc-35a9-4acc-84f3-6b437bb03925-plugins-config-volume\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:34.078504 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.078473    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3e574fc-35a9-4acc-84f3-6b437bb03925" containerID="a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744" exitCode=0
Sep 10 15:04:34.078954 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.078516    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"
Sep 10 15:04:34.078954 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.078528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg" event={"ID":"e3e574fc-35a9-4acc-84f3-6b437bb03925","Type":"ContainerDied","Data":"a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744"}
Sep 10 15:04:34.078954 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.078566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg" event={"ID":"e3e574fc-35a9-4acc-84f3-6b437bb03925","Type":"ContainerDied","Data":"f240c84e03c1f184c78af6ea64d5a59782956cc1657bab513f335417d408cbd0"}
Sep 10 15:04:34.078954 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.078587    2566 scope.go:117] "RemoveContainer" containerID="a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744"
Sep 10 15:04:34.087633 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.087614    2566 scope.go:117] "RemoveContainer" containerID="a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744"
Sep 10 15:04:34.087942 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:04:34.087925    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744\": container with ID starting with a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744 not found: ID does not exist" containerID="a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744"
Sep 10 15:04:34.087994 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.087949    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744"} err="failed to get container status \"a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744\": rpc error: code = NotFound desc = could not find container \"a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744\": container with ID starting with a2a0b9f45032d43d838d7255eac7e8f20e49880f5300b6104242e25618e24744 not found: ID does not exist"
Sep 10 15:04:34.110322 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.110298    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7739ee30-af7f-4a83-8450-38b99d3faf26" path="/var/lib/kubelet/pods/7739ee30-af7f-4a83-8450-38b99d3faf26/volumes"
Sep 10 15:04:34.119753 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.119731    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"]
Sep 10 15:04:34.135843 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:34.135818    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-12b6f6-765f8699f4-rmjlg"]
Sep 10 15:04:36.109532 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:36.109493    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e574fc-35a9-4acc-84f3-6b437bb03925" path="/var/lib/kubelet/pods/e3e574fc-35a9-4acc-84f3-6b437bb03925/volumes"
Sep 10 15:04:52.103835 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.103802    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"]
Sep 10 15:04:52.104314 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.104002    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w" podUID="9c32d6e0-b1da-421b-9f36-d714627a81aa" containerName="maas-api" containerID="cri-o://d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f" gracePeriod=30
Sep 10 15:04:52.344185 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.344154    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 15:04:52.476222 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.476193    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-metrics-certs\") pod \"9c32d6e0-b1da-421b-9f36-d714627a81aa\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") "
Sep 10 15:04:52.476379 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.476243    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnq5m\" (UniqueName: \"kubernetes.io/projected/9c32d6e0-b1da-421b-9f36-d714627a81aa-kube-api-access-tnq5m\") pod \"9c32d6e0-b1da-421b-9f36-d714627a81aa\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") "
Sep 10 15:04:52.476379 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.476298    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls\") pod \"9c32d6e0-b1da-421b-9f36-d714627a81aa\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") "
Sep 10 15:04:52.476379 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.476318    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32d6e0-b1da-421b-9f36-d714627a81aa-openshift-service-ca\") pod \"9c32d6e0-b1da-421b-9f36-d714627a81aa\" (UID: \"9c32d6e0-b1da-421b-9f36-d714627a81aa\") "
Sep 10 15:04:52.476756 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.476730    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c32d6e0-b1da-421b-9f36-d714627a81aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c32d6e0-b1da-421b-9f36-d714627a81aa" (UID: "9c32d6e0-b1da-421b-9f36-d714627a81aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:04:52.478379 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.478342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c32d6e0-b1da-421b-9f36-d714627a81aa" (UID: "9c32d6e0-b1da-421b-9f36-d714627a81aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:04:52.478475 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.478390    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c32d6e0-b1da-421b-9f36-d714627a81aa" (UID: "9c32d6e0-b1da-421b-9f36-d714627a81aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:04:52.478475 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.478443    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c32d6e0-b1da-421b-9f36-d714627a81aa-kube-api-access-tnq5m" (OuterVolumeSpecName: "kube-api-access-tnq5m") pod "9c32d6e0-b1da-421b-9f36-d714627a81aa" (UID: "9c32d6e0-b1da-421b-9f36-d714627a81aa"). InnerVolumeSpecName "kube-api-access-tnq5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:52.577589 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.577564    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnq5m\" (UniqueName: \"kubernetes.io/projected/9c32d6e0-b1da-421b-9f36-d714627a81aa-kube-api-access-tnq5m\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:52.577589 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.577587    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:52.577756 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.577596    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32d6e0-b1da-421b-9f36-d714627a81aa-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:52.577756 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:52.577604    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c32d6e0-b1da-421b-9f36-d714627a81aa-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:04:53.163847 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.163806    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c32d6e0-b1da-421b-9f36-d714627a81aa" containerID="d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f" exitCode=0
Sep 10 15:04:53.163847 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.163848    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"
Sep 10 15:04:53.164384 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.163894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w" event={"ID":"9c32d6e0-b1da-421b-9f36-d714627a81aa","Type":"ContainerDied","Data":"d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f"}
Sep 10 15:04:53.164384 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.163945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w" event={"ID":"9c32d6e0-b1da-421b-9f36-d714627a81aa","Type":"ContainerDied","Data":"9fafb7e0f8a034b152c08e8d39a6aaf28a356d6ccb39dce3d6634d64ee9cd22d"}
Sep 10 15:04:53.164384 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.163972    2566 scope.go:117] "RemoveContainer" containerID="d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f"
Sep 10 15:04:53.173525 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.173508    2566 scope.go:117] "RemoveContainer" containerID="d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f"
Sep 10 15:04:53.173818 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:04:53.173797    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f\": container with ID starting with d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f not found: ID does not exist" containerID="d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f"
Sep 10 15:04:53.173869 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.173829    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f"} err="failed to get container status \"d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f\": rpc error: code = NotFound desc = could not find container \"d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f\": container with ID starting with d1f57e7a5df9849f40f0df3c747729b533eef0daf84fab1ad360e5bd6e134f1f not found: ID does not exist"
Sep 10 15:04:53.209460 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.209437    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"]
Sep 10 15:04:53.223299 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:53.223275    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e6941c-75dbcbf487-7dx5w"]
Sep 10 15:04:54.110145 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:04:54.110101    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c32d6e0-b1da-421b-9f36-d714627a81aa" path="/var/lib/kubelet/pods/9c32d6e0-b1da-421b-9f36-d714627a81aa/volumes"
Sep 10 15:06:54.390025 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:06:54.389939    2566 scope.go:117] "RemoveContainer" containerID="c87c13fc14b30fa0cc891dacb85389f7fba8a99bcf55398c10b738fb7ea22f13"
Sep 10 15:07:54.160809 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:07:54.160782    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 15:07:54.164951 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:07:54.164921    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 15:07:54.166579 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:07:54.166558    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 15:07:54.170504 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:07:54.170483    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hbrcr_055abf0a-a771-42fa-aa0a-be5cd2b73692/ovn-acl-logging/0.log"
Sep 10 15:10:40.633434 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633392    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"]
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633708    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fce4160-4842-4a5f-be43-09e75d504ea3" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633721    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fce4160-4842-4a5f-be43-09e75d504ea3" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633733    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e574fc-35a9-4acc-84f3-6b437bb03925" containerName="payload-pre-processing"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633739    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e574fc-35a9-4acc-84f3-6b437bb03925" containerName="payload-pre-processing"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633750    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c05ce102-65da-4ceb-ab97-eaa1aff76a3e" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633756    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c05ce102-65da-4ceb-ab97-eaa1aff76a3e" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633763    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c71753-9692-454b-8477-9a0d17eb7a64" containerName="payload-processing"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633769    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c71753-9692-454b-8477-9a0d17eb7a64" containerName="payload-processing"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633782    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c32d6e0-b1da-421b-9f36-d714627a81aa" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633787    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c32d6e0-b1da-421b-9f36-d714627a81aa" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633797    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1252c0f0-f065-4f4c-a98a-2de4ecfa531b" containerName="payload-processing"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633803    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1252c0f0-f065-4f4c-a98a-2de4ecfa531b" containerName="payload-processing"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633850    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fce4160-4842-4a5f-be43-09e75d504ea3" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633857    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c71753-9692-454b-8477-9a0d17eb7a64" containerName="payload-processing"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633863    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c32d6e0-b1da-421b-9f36-d714627a81aa" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633868    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e574fc-35a9-4acc-84f3-6b437bb03925" containerName="payload-pre-processing"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633876    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c05ce102-65da-4ceb-ab97-eaa1aff76a3e" containerName="maas-api"
Sep 10 15:10:40.633980 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.633882    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1252c0f0-f065-4f4c-a98a-2de4ecfa531b" containerName="payload-processing"
Sep 10 15:10:40.636941 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.636919    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.640049 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.640012    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:10:40.640212 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.640017    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:10:40.640212 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.640017    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7j4nl\""
Sep 10 15:10:40.640342 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.640027    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:10:40.640342 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.640026    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 15:10:40.647252 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.647226    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"]
Sep 10 15:10:40.730703 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.730669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgds\" (UniqueName: \"kubernetes.io/projected/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-kube-api-access-9rgds\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.730861 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.730752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-maas-api-tls\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.730861 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.730784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-metrics-certs\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.730968 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.730866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-openshift-service-ca\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.831444 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.831411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-metrics-certs\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.831589 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.831472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-openshift-service-ca\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.831589 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.831500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgds\" (UniqueName: \"kubernetes.io/projected/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-kube-api-access-9rgds\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.831589 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.831542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-maas-api-tls\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.832155 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.832124    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-openshift-service-ca\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.833914 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.833893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-maas-api-tls\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.834030 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.833963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-metrics-certs\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.839574 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.839556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgds\" (UniqueName: \"kubernetes.io/projected/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-kube-api-access-9rgds\") pod \"maas-api-59bbbd559c-rb4j7\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") " pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:40.949309 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:40.949277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:41.218448 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:41.218423    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"]
Sep 10 15:10:41.220912 ip-10-0-129-162 kubenswrapper[2566]: W0910 15:10:41.220886    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd31e01_67fd_4e3e_8869_67dd5f3a161b.slice/crio-600a9aba4008e3566b7b17295ae94cc83e8915bb36448b26076dfd44affd3a75 WatchSource:0}: Error finding container 600a9aba4008e3566b7b17295ae94cc83e8915bb36448b26076dfd44affd3a75: Status 404 returned error can't find the container with id 600a9aba4008e3566b7b17295ae94cc83e8915bb36448b26076dfd44affd3a75
Sep 10 15:10:41.222266 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:41.222248    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:10:41.628624 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:41.628543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7" event={"ID":"2dd31e01-67fd-4e3e-8869-67dd5f3a161b","Type":"ContainerStarted","Data":"600a9aba4008e3566b7b17295ae94cc83e8915bb36448b26076dfd44affd3a75"}
Sep 10 15:10:42.635096 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:42.635055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7" event={"ID":"2dd31e01-67fd-4e3e-8869-67dd5f3a161b","Type":"ContainerStarted","Data":"14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9"}
Sep 10 15:10:42.635512 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:42.635214    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:42.652749 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:42.652480    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7" podStartSLOduration=2.055705328 podStartE2EDuration="2.652462108s" totalImagesPullingTime="596.75678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:10:42.651871751 +0000 UTC m=+2569.077609172" watchObservedRunningTime="2026-09-10 15:10:42.652462108 +0000 UTC m=+2569.078199528"
Sep 10 15:10:48.129227 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.129192    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"]
Sep 10 15:10:48.129655 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.129371    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7" podUID="2dd31e01-67fd-4e3e-8869-67dd5f3a161b" containerName="maas-api" containerID="cri-o://14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9" gracePeriod=30
Sep 10 15:10:48.134496 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.134473    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:48.165510 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.165475    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"]
Sep 10 15:10:48.168579 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.168560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.178282 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.178260    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"]
Sep 10 15:10:48.181933 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.181901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26qw\" (UniqueName: \"kubernetes.io/projected/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-kube-api-access-t26qw\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.182036 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.181943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-metrics-certs\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.182036 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.181996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-maas-api-tls\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.182036 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.182026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-openshift-service-ca\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.282905 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.282872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-metrics-certs\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.283066 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.282914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-maas-api-tls\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.283066 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.282935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-openshift-service-ca\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.283066 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.282979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26qw\" (UniqueName: \"kubernetes.io/projected/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-kube-api-access-t26qw\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.283599 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.283577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-openshift-service-ca\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.285260 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.285236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-metrics-certs\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.285500 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.285475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-maas-api-tls\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.295660 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.295632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26qw\" (UniqueName: \"kubernetes.io/projected/405b5caf-81ee-4a88-80e5-d6d8c4c47c42-kube-api-access-t26qw\") pod \"maas-api-79b7b5b657-ksrvc\" (UID: \"405b5caf-81ee-4a88-80e5-d6d8c4c47c42\") " pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.354282 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.354261    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:48.383937 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.383859    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-maas-api-tls\") pod \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") "
Sep 10 15:10:48.383937 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.383905    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-metrics-certs\") pod \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") "
Sep 10 15:10:48.383937 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.383932    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rgds\" (UniqueName: \"kubernetes.io/projected/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-kube-api-access-9rgds\") pod \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") "
Sep 10 15:10:48.384163 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.383959    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-openshift-service-ca\") pod \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\" (UID: \"2dd31e01-67fd-4e3e-8869-67dd5f3a161b\") "
Sep 10 15:10:48.384350 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.384324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2dd31e01-67fd-4e3e-8869-67dd5f3a161b" (UID: "2dd31e01-67fd-4e3e-8869-67dd5f3a161b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:10:48.385807 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.385769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2dd31e01-67fd-4e3e-8869-67dd5f3a161b" (UID: "2dd31e01-67fd-4e3e-8869-67dd5f3a161b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:10:48.386569 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.386529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-kube-api-access-9rgds" (OuterVolumeSpecName: "kube-api-access-9rgds") pod "2dd31e01-67fd-4e3e-8869-67dd5f3a161b" (UID: "2dd31e01-67fd-4e3e-8869-67dd5f3a161b"). InnerVolumeSpecName "kube-api-access-9rgds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:10:48.387311 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.387289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2dd31e01-67fd-4e3e-8869-67dd5f3a161b" (UID: "2dd31e01-67fd-4e3e-8869-67dd5f3a161b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:10:48.485548 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.485516    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-maas-api-tls\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:10:48.485548 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.485548    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-metrics-certs\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:10:48.485680 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.485558    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgds\" (UniqueName: \"kubernetes.io/projected/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-kube-api-access-9rgds\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:10:48.485680 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.485568    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd31e01-67fd-4e3e-8869-67dd5f3a161b-openshift-service-ca\") on node \"ip-10-0-129-162.ec2.internal\" DevicePath \"\""
Sep 10 15:10:48.509657 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.509634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:48.663923 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.663891    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dd31e01-67fd-4e3e-8869-67dd5f3a161b" containerID="14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9" exitCode=0
Sep 10 15:10:48.664072 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.663938    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"
Sep 10 15:10:48.664072 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.663979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7" event={"ID":"2dd31e01-67fd-4e3e-8869-67dd5f3a161b","Type":"ContainerDied","Data":"14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9"}
Sep 10 15:10:48.664072 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.664039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7" event={"ID":"2dd31e01-67fd-4e3e-8869-67dd5f3a161b","Type":"ContainerDied","Data":"600a9aba4008e3566b7b17295ae94cc83e8915bb36448b26076dfd44affd3a75"}
Sep 10 15:10:48.664072 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.664070    2566 scope.go:117] "RemoveContainer" containerID="14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9"
Sep 10 15:10:48.672753 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.672609    2566 scope.go:117] "RemoveContainer" containerID="14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9"
Sep 10 15:10:48.673192 ip-10-0-129-162 kubenswrapper[2566]: E0910 15:10:48.673144    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9\": container with ID starting with 14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9 not found: ID does not exist" containerID="14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9"
Sep 10 15:10:48.673301 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.673183    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9"} err="failed to get container status \"14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9\": rpc error: code = NotFound desc = could not find container \"14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9\": container with ID starting with 14401329ed6d564105de61f2c053bb267363f06c3915a3d31e6df75c17c276b9 not found: ID does not exist"
Sep 10 15:10:48.685426 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.685403    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"]
Sep 10 15:10:48.689198 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.689177    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59bbbd559c-rb4j7"]
Sep 10 15:10:48.785503 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:48.785181    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"]
Sep 10 15:10:48.789295 ip-10-0-129-162 kubenswrapper[2566]: W0910 15:10:48.789258    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405b5caf_81ee_4a88_80e5_d6d8c4c47c42.slice/crio-14f2c987e07b109ed56b4b727d123a90468fb749284783a2e842dc3a16795839 WatchSource:0}: Error finding container 14f2c987e07b109ed56b4b727d123a90468fb749284783a2e842dc3a16795839: Status 404 returned error can't find the container with id 14f2c987e07b109ed56b4b727d123a90468fb749284783a2e842dc3a16795839
Sep 10 15:10:49.670200 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:49.670165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc" event={"ID":"405b5caf-81ee-4a88-80e5-d6d8c4c47c42","Type":"ContainerStarted","Data":"17ef9667c5385726b14bd2b9dd06afb8c6722daa0103af19327bcc33109d7a07"}
Sep 10 15:10:49.670635 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:49.670226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc" event={"ID":"405b5caf-81ee-4a88-80e5-d6d8c4c47c42","Type":"ContainerStarted","Data":"14f2c987e07b109ed56b4b727d123a90468fb749284783a2e842dc3a16795839"}
Sep 10 15:10:49.670635 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:49.670258    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:10:49.689981 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:49.689932    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc" podStartSLOduration=1.386666578 podStartE2EDuration="1.689916937s" totalImagesPullingTime="303.250359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:10:49.688135758 +0000 UTC m=+2576.113873179" watchObservedRunningTime="2026-09-10 15:10:49.689916937 +0000 UTC m=+2576.115654357"
Sep 10 15:10:50.109730 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:50.109642    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd31e01-67fd-4e3e-8869-67dd5f3a161b" path="/var/lib/kubelet/pods/2dd31e01-67fd-4e3e-8869-67dd5f3a161b/volumes"
Sep 10 15:10:55.681453 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:10:55.681416    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79b7b5b657-ksrvc"
Sep 10 15:11:18.370491 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:18.370418    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77dd699645-w7fqh_b513491b-fb81-4782-ae5a-1cf7305e88c3/copy-manifests/0.log"
Sep 10 15:11:18.384936 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:18.384904    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77dd699645-w7fqh_b513491b-fb81-4782-ae5a-1cf7305e88c3/manager/0.log"
Sep 10 15:11:18.865521 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:18.865487    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79b7b5b657-ksrvc_405b5caf-81ee-4a88-80e5-d6d8c4c47c42/maas-api/0.log"
Sep 10 15:11:18.981160 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:18.981135    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817525-gmnqj_bf124b00-7259-4e7d-aa67-0b6072adcbd6/cleanup/0.log"
Sep 10 15:11:21.261908 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:21.261880    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-29hxg_df48f725-b7ab-46c1-8941-0eb09a65fbdc/discovery/0.log"
Sep 10 15:11:21.377566 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:21.377514    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c68946dcf-7wmsv_aa103c51-36e2-4470-b069-322f7b5fdc25/kube-auth-proxy/0.log"
Sep 10 15:11:32.178703 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:32.178650    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8tb9v_6390be7d-6bcc-44e6-be5b-7d4f3fdd92e4/global-pull-secret-syncer/0.log"
Sep 10 15:11:32.343438 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:32.343405    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sxhmb_47492653-52e9-41d3-94e3-616fef6d8780/konnectivity-agent/0.log"
Sep 10 15:11:32.363463 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:32.363439    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-162.ec2.internal_d0c71d27d7840b6e3fe5cea3210406d3/haproxy/0.log"
Sep 10 15:11:39.747679 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:39.747654    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-lnvg9_ee67ac4c-3579-4d26-a029-9f782d31589f/cluster-monitoring-operator/0.log"
Sep 10 15:11:39.996156 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:39.996130    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nsdvn_61536883-dc4c-4599-b76d-0b2cbca31580/node-exporter/0.log"
Sep 10 15:11:40.016601 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:40.016580    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nsdvn_61536883-dc4c-4599-b76d-0b2cbca31580/kube-rbac-proxy/0.log"
Sep 10 15:11:40.040802 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:40.040756    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nsdvn_61536883-dc4c-4599-b76d-0b2cbca31580/init-textfile/0.log"
Sep 10 15:11:42.780301 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.780273    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/1.log"
Sep 10 15:11:42.787471 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.787439    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bfwnd_2d4f3935-7fa7-4cd5-aa0b-a5178e683e41/console-operator/2.log"
Sep 10 15:11:42.814954 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.814926    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"]
Sep 10 15:11:42.815210 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.815197    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd31e01-67fd-4e3e-8869-67dd5f3a161b" containerName="maas-api"
Sep 10 15:11:42.815266 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.815210    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd31e01-67fd-4e3e-8869-67dd5f3a161b" containerName="maas-api"
Sep 10 15:11:42.815309 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.815269    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd31e01-67fd-4e3e-8869-67dd5f3a161b" containerName="maas-api"
Sep 10 15:11:42.818245 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.818227    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.820446 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.820421    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fqt5c\"/\"openshift-service-ca.crt\""
Sep 10 15:11:42.820574 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.820493    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fqt5c\"/\"kube-root-ca.crt\""
Sep 10 15:11:42.820574 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.820528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fqt5c\"/\"default-dockercfg-f4hgn\""
Sep 10 15:11:42.826202 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.826175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"]
Sep 10 15:11:42.872428 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.872404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-lib-modules\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.872528 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.872438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-proc\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.872528 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.872471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-sys\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.872604 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.872532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-podres\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.872604 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.872575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jp68\" (UniqueName: \"kubernetes.io/projected/ae4e1b13-4d46-446a-9ced-a24d64204507-kube-api-access-7jp68\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.973896 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.973859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-lib-modules\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.973896 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.973903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-proc\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.974053 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.973924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-sys\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.974053 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.973940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-podres\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.974053 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.973966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jp68\" (UniqueName: \"kubernetes.io/projected/ae4e1b13-4d46-446a-9ced-a24d64204507-kube-api-access-7jp68\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.974053 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.973993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-proc\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.974053 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.974000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-sys\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.974053 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.974041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-lib-modules\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.974256 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.974081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae4e1b13-4d46-446a-9ced-a24d64204507-podres\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:42.981818 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:42.981788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jp68\" (UniqueName: \"kubernetes.io/projected/ae4e1b13-4d46-446a-9ced-a24d64204507-kube-api-access-7jp68\") pod \"perf-node-gather-daemonset-wn8d4\" (UID: \"ae4e1b13-4d46-446a-9ced-a24d64204507\") " pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:43.129389 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:43.129334    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:43.402429 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:43.402404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"]
Sep 10 15:11:43.404493 ip-10-0-129-162 kubenswrapper[2566]: W0910 15:11:43.404468    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podae4e1b13_4d46_446a_9ced_a24d64204507.slice/crio-3c38faf18052553b0bb12ad08367c03bff0587299e12d5bb146ed0746f37ed05 WatchSource:0}: Error finding container 3c38faf18052553b0bb12ad08367c03bff0587299e12d5bb146ed0746f37ed05: Status 404 returned error can't find the container with id 3c38faf18052553b0bb12ad08367c03bff0587299e12d5bb146ed0746f37ed05
Sep 10 15:11:43.913954 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:43.913905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4" event={"ID":"ae4e1b13-4d46-446a-9ced-a24d64204507","Type":"ContainerStarted","Data":"a9ca490cddbdffddf0527fa03979d543d5674e9e1c1bf83f07ee38a28440f775"}
Sep 10 15:11:43.914305 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:43.913976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4" event={"ID":"ae4e1b13-4d46-446a-9ced-a24d64204507","Type":"ContainerStarted","Data":"3c38faf18052553b0bb12ad08367c03bff0587299e12d5bb146ed0746f37ed05"}
Sep 10 15:11:43.914305 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:43.914061    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:43.945600 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:43.945557    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4" podStartSLOduration=1.945544297 podStartE2EDuration="1.945544297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:43.94481881 +0000 UTC m=+2630.370556229" watchObservedRunningTime="2026-09-10 15:11:43.945544297 +0000 UTC m=+2630.371281718"
Sep 10 15:11:44.218106 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:44.218040    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-djwzm_0538b272-5201-49b4-bff0-69c7146973ab/volume-data-source-validator/0.log"
Sep 10 15:11:45.346036 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:45.346008    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98m8q_cd04e0bc-77f9-44c0-87a9-d8123246c487/dns/0.log"
Sep 10 15:11:45.367383 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:45.367359    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98m8q_cd04e0bc-77f9-44c0-87a9-d8123246c487/kube-rbac-proxy/0.log"
Sep 10 15:11:45.483579 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:45.483554    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5n69_352316f0-970c-43ad-befe-ffeaaad7e582/dns-node-resolver/0.log"
Sep 10 15:11:46.048562 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:46.048534    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fvv6t_532db11a-6291-4687-8d65-e71384f16400/node-ca/0.log"
Sep 10 15:11:47.281872 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:47.281840    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-29hxg_df48f725-b7ab-46c1-8941-0eb09a65fbdc/discovery/0.log"
Sep 10 15:11:47.316434 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:47.316412    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c68946dcf-7wmsv_aa103c51-36e2-4470-b069-322f7b5fdc25/kube-auth-proxy/0.log"
Sep 10 15:11:48.556763 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:48.556735    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p26lr_312e6872-7076-44d1-8bbc-0c2b9b4161d5/serve-healthcheck-canary/0.log"
Sep 10 15:11:49.170479 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:49.170449    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-phvc7_edc97f05-11d3-4259-803c-2e7cb859844e/kube-rbac-proxy/0.log"
Sep 10 15:11:49.193758 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:49.193732    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-phvc7_edc97f05-11d3-4259-803c-2e7cb859844e/exporter/0.log"
Sep 10 15:11:49.216604 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:49.216581    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-phvc7_edc97f05-11d3-4259-803c-2e7cb859844e/extractor/0.log"
Sep 10 15:11:49.928735 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:49.928672    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fqt5c/perf-node-gather-daemonset-wn8d4"
Sep 10 15:11:52.094578 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:52.094548    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77dd699645-w7fqh_b513491b-fb81-4782-ae5a-1cf7305e88c3/manager/0.log"
Sep 10 15:11:52.114030 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:52.114009    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77dd699645-w7fqh_b513491b-fb81-4782-ae5a-1cf7305e88c3/copy-manifests/0.log"
Sep 10 15:11:53.625950 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:53.625922    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c4b8b8b58-88xlh_eeb78ed6-059d-47a7-a246-f78fbf5299a6/manager/0.log"
Sep 10 15:11:53.692747 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:53.691151    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-swrr5_b5e80416-fa13-4949-864e-0d14149e7d70/openshift-lws-operator/0.log"
Sep 10 15:11:53.833301 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:53.833277    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-29hxg_df48f725-b7ab-46c1-8941-0eb09a65fbdc/discovery/0.log"
Sep 10 15:11:53.847485 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:53.847462    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c68946dcf-7wmsv_aa103c51-36e2-4470-b069-322f7b5fdc25/kube-auth-proxy/0.log"
Sep 10 15:11:55.817372 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:55.817344    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-29hxg_df48f725-b7ab-46c1-8941-0eb09a65fbdc/discovery/0.log"
Sep 10 15:11:55.831267 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:55.831243    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c68946dcf-7wmsv_aa103c51-36e2-4470-b069-322f7b5fdc25/kube-auth-proxy/0.log"
Sep 10 15:11:59.415463 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:59.415433    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-f4rj9_7db503b0-a2c2-415b-9481-6b65ad9b7d23/kube-storage-version-migrator-operator/1.log"
Sep 10 15:11:59.416439 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:11:59.416418    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-f4rj9_7db503b0-a2c2-415b-9481-6b65ad9b7d23/kube-storage-version-migrator-operator/0.log"
Sep 10 15:12:00.626670 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:00.626644    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpg6r_538b5488-3d52-4ae3-8ffc-5e3be2ac32de/kube-multus-additional-cni-plugins/0.log"
Sep 10 15:12:00.648487 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:00.648463    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpg6r_538b5488-3d52-4ae3-8ffc-5e3be2ac32de/egress-router-binary-copy/0.log"
Sep 10 15:12:00.671311 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:00.671284    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpg6r_538b5488-3d52-4ae3-8ffc-5e3be2ac32de/cni-plugins/0.log"
Sep 10 15:12:00.693504 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:00.693479    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpg6r_538b5488-3d52-4ae3-8ffc-5e3be2ac32de/bond-cni-plugin/0.log"
Sep 10 15:12:00.713912 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:00.713894    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpg6r_538b5488-3d52-4ae3-8ffc-5e3be2ac32de/routeoverride-cni/0.log"
Sep 10 15:12:00.735606 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:00.735589    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpg6r_538b5488-3d52-4ae3-8ffc-5e3be2ac32de/whereabouts-cni-bincopy/0.log"
Sep 10 15:12:00.757971 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:00.757951    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpg6r_538b5488-3d52-4ae3-8ffc-5e3be2ac32de/whereabouts-cni/0.log"
Sep 10 15:12:01.004589 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:01.004564    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jr8gf_3d97cd1e-f9fb-4f92-ab4c-2068152f5d01/kube-multus/0.log"
Sep 10 15:12:01.135321 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:01.135299    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qfbg6_fe629083-72c1-416f-a8ab-38fe211e7598/network-metrics-daemon/0.log"
Sep 10 15:12:01.157850 ip-10-0-129-162 kubenswrapper[2566]: I0910 15:12:01.157831    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qfbg6_fe629083-72c1-416f-a8ab-38fe211e7598/kube-rbac-proxy/0.log"