Jul 28 17:08:49.205663 ip-10-0-129-222 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:08:49.619397 ip-10-0-129-222 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:08:49.619397 ip-10-0-129-222 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:08:49.619397 ip-10-0-129-222 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:08:49.619397 ip-10-0-129-222 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:08:49.619397 ip-10-0-129-222 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:08:49.621443 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.621331    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:08:49.627066 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627045    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:08:49.627066 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627065    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:08:49.627066 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627069    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627072    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627077    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627080    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627083    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627086    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627089    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627092    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627095    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627097    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627100    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627109    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627112    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627114    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627117    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627120    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627122    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627126    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627128    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627131    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:08:49.627187 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627134    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627136    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627152    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627156    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627159    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627162    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627164    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627167    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627169    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627172    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627174    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627177    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627180    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627183    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627185    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627188    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627191    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627193    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627196    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627199    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:08:49.627662 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627202    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627205    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627207    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627210    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627218    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627221    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627223    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627226    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627228    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627231    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627234    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627236    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627238    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627241    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627244    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627272    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627278    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627281    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627285    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:08:49.628359 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627287    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627290    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627293    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627296    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627299    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627303    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627306    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627309    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627312    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627314    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627317    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627319    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627322    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627325    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627356    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627360    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627363    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627366    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627369    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627372    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:08:49.629103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627375    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627377    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627380    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627383    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627387    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627390    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627393    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627396    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627399    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627401    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627404    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627406    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627408    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627411    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627413    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627417    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627421    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:08:49.629729 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.627425    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629814    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629829    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629833    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629836    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629840    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629843    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629846    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629849    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629851    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629854    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629857    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629859    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629862    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629865    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629868    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629871    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629874    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629876    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:08:49.630144 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629879    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629881    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629884    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629888    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629890    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629892    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629900    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629902    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629905    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629908    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629910    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629913    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629915    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629918    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629920    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629924    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629927    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629931    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629936    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:08:49.630641 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629940    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629944    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629948    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629952    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629956    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629968    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629972    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629975    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629977    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629980    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629983    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629985    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629988    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629991    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629993    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629996    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.629998    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630001    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630003    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630007    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:08:49.631118 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630009    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630012    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630015    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630017    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630020    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630022    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630025    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630029    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630032    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630035    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630038    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630041    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630044    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630047    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630049    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630053    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630056    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630058    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630061    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630063    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:08:49.631640 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630066    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630069    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630071    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630074    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630076    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630079    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630083    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630086    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630088    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630091    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630093    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630096    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630098    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630100    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630103    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630106    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630108    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630111    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630114    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:08:49.632125 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630117    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630120    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630123    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630604    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630611    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630615    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630618    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630621    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630624    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630627    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630630    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630632    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630635    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630638    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630641    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630643    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630646    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630649    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630652    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630655    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:08:49.632611 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630658    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630661    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630664    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630666    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630669    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630671    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630674    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630676    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630679    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630681    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630684    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630687    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630690    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630693    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630698    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630701    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630705    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630708    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630711    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630713    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:08:49.633139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630716    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630720    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630722    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630725    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630727    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630730    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630732    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630735    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630737    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630740    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630743    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630745    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630749    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630751    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630754    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630756    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630759    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630762    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630764    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:08:49.633648 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630768    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630771    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630774    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630777    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630780    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630783    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630785    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630788    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630790    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630793    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630795    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630798    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630800    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630804    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630808    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630810    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630813    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630815    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630818    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:08:49.634103 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630821    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630823    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630825    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630828    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630830    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630833    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630835    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630838    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630841    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630843    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630846    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630848    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630852    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630854    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630857    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630860    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630862    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630865    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630868    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630871    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:08:49.634646 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630874    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630876    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630879    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.630882    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.630959    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.630970    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.630981    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.630997    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631004    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631009    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631014    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631018    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631021    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631024    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631028    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631031    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631035    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631038    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631041    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631044    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631048    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631052    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631057    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631060    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:08:49.635106 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631063    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631066    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631069    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631074    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631077    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631080    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631084    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631087    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631090    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631093    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631097    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631100    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631104    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631107    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631110    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631113    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631116    2560 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631119    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631124    2560 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631127    2560 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631130    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631134    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631137    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631141    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:08:49.635729 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631144    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631147    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631150    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631153    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631156    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631160    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631163    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631166    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631169    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631172    2560 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631176    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631179    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631183    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631186    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631189    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631192    2560 flags.go:64] FLAG: --help="false"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631195    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-129-222.ec2.internal"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631199    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631202    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631205    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631208    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631212    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631214    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:08:49.636280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631217    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631220    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631223    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631225    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631229    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631231    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631234    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631237    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631240    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631243    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631245    2560 flags.go:64] FLAG: --lock-file=""
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631249    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631252    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631255    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631260    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631264    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631266    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631269    2560 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631272    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631275    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631278    2560 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631281    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631286    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631289    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631293    2560 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:08:49.636861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631296    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631299    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631302    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631305    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631308    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631311    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631314    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631322    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631325    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631328    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631330    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631333    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631356    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631360    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631365    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631368    2560 flags.go:64] FLAG: --port="10250"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631371    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631376    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-090e5c69bf15e78f3"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631380    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631383    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631386    2560 flags.go:64] FLAG: --register-node="true"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631389    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631393    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631396    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:08:49.637586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631400    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631403    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631406    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631415    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631418    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631421    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631424    2560 flags.go:64] FLAG: --runonce="false"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631427    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631430    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631433    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631436    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631439    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631442    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631445    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631448    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631451    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631454    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631457    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631460    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631462    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631465    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631468    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631474    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631477    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631480    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:08:49.638195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631486    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631490    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631493    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631495    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631498    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631502    2560 flags.go:64] FLAG: --v="2"
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631511    2560 flags.go:64] FLAG: --version="false"
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631516    2560 flags.go:64] FLAG: --vmodule=""
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631520    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.631523    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631662    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631667    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631670    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631674    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631677    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631680    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631687    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631690    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631693    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631696    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631699    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631701    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:08:49.638826 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631704    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631706    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631709    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631712    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631715    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631717    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631720    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631723    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631726    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631728    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631733    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631735    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631738    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631741    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631743    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631746    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631749    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631752    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631758    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:08:49.639388 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631761    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631777    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631780    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631782    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631785    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631787    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631790    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631792    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631795    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631797    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631800    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631803    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631805    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631808    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631810    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631813    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631815    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631818    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631820    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631823    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:08:49.639840 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631826    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631828    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631831    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631834    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631837    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631839    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631842    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631844    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631847    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631850    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631852    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631855    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631858    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631862    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631864    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631867    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631870    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631872    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631875    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631877    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:08:49.640315 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631880    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631882    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631885    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631887    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631890    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631892    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631895    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631897    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631900    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631903    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631905    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631907    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631910    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631914    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631917    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631922    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631925    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631927    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631930    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631933    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:08:49.640803 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631936    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631938    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631942    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631946    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631950    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631955    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631958    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.631961    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.632462    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.639211    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.639229    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.639250    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.639256    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639311    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:08:49.641271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639316    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639319    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639322    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639325    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639327    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639330    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639333    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639335    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639338    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639353    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639357    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639361    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639365    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639368    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639372    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639375    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639377    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639380    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639383    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639387    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:08:49.641727 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639390    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639393    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639395    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639398    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639401    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639404    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639406    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639410    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639417    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639421    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639423    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639426    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639428    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639432    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639436    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639439    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639442    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639445    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639448    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639451    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:08:49.642202 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639453    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639457    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639459    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639462    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639464    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639467    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639469    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639472    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639475    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639477    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639480    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639482    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639485    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639487    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639490    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639493    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639495    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639498    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639500    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:08:49.642714 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639503    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639506    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639509    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639512    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639514    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639516    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639519    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639522    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639524    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639527    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639530    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639533    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639535    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639538    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639540    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639543    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639545    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639547    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639550    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639553    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:08:49.643192 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639555    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639558    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639560    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639563    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639565    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639568    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639571    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639573    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639577    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639580    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639583    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639586    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639588    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639591    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639594    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639597    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639599    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639602    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:08:49.643769 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639604    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.639610    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639723    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639727    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639731    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639734    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639736    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639739    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639742    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639744    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639746    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639749    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639752    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639754    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639757    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:08:49.644275 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639759    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639762    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639764    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639767    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639769    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639773    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639775    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639778    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639780    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639783    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639785    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639788    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639790    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639793    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639796    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639799    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639801    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639804    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639807    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639809    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:08:49.644696 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639811    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639814    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639817    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639819    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639822    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639824    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639827    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639829    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639832    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639835    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639839    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639842    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639844    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639848    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639851    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639854    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639858    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639861    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639864    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:08:49.645232 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639867    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639869    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639872    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639874    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639877    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639880    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639882    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639885    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639888    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639891    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639893    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639896    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639898    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639901    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639904    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639906    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639909    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639911    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639914    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:08:49.645771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639916    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639919    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639921    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639924    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639926    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639929    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639931    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639935    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639938    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639941    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639944    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639947    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639949    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639952    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639954    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639957    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639959    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639962    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639964    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:08:49.646526 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639967    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639970    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639972    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639975    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639978    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639980    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639983    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639986    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:49.639988    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.639993    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.640119    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.642642    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.644013    2560 server.go:1019] "Starting client certificate rotation"
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.644132    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:08:49.647163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.644230    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:08:49.647537 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.646195    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:08:49.647537 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.647396    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:08:49.656970 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.656953    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:08:49.658970 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.658958    2560 log.go:25] "Validated CRI v1 image API"
Jul 28 17:08:49.661230 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.661217    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:08:49.664437 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.664414    2560 fs.go:135] Filesystem UUIDs: map[2ef98045-f96f-4bd7-9486-74038aa92a0e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f2b181c4-71ca-415d-bba8-5bcf1677e9fc:/dev/nvme0n1p3]
Jul 28 17:08:49.664529 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.664435    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:08:49.669626 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.669504    2560 manager.go:216] Machine: {Timestamp:2026-07-28 17:08:49.668147617 +0000 UTC m=+0.340475528 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3228114 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e69e576a9e29f3032eaa212542f44 SystemUUID:ec2e69e5-76a9-e29f-3032-eaa212542f44 BootID:7816fac3-9b21-4abf-ab85-170a0208731c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:5e:d5:bc:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:5e:d5:bc:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:19:08:24:a7:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:08:49.669626 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.669618    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:08:49.669784 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.669767    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:08:49.670705 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.670682    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:08:49.670871 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.670712    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:08:49.671721 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.671709    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:08:49.671753 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.671724    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:08:49.671753 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.671732    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:08:49.672442 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.672432    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:08:49.673189 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.673179    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:08:49.676499 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.676487    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:08:49.676625 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.676615    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:08:49.677479 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.677463    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:08:49.677520 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.677496    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:08:49.677520 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.677515    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:08:49.677587 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.677524    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:08:49.677587 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.677533    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:08:49.677876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.677865    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:08:49.677913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.677884    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:08:49.680335 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.680319    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:08:49.682190 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.682174    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:08:49.682277 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.682215    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:08:49.682277 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.682265    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:08:49.683604 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683590    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:08:49.683679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683610    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:08:49.683679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683620    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:08:49.683679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683628    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:08:49.683679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683641    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:08:49.683679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683649    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:08:49.683679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683661    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:08:49.683679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683669    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:08:49.683679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683679    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:08:49.683910 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683688    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:08:49.683910 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683700    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:08:49.683910 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.683709    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:08:49.685046 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.685034    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:08:49.685115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.685047    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:08:49.688301 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.688286    2560 server.go:1268] "Started kubelet"
Jul 28 17:08:49.688421 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.688364    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:08:49.688468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.688408    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:08:49.688504 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.688471    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:08:49.688946 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.688930    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:08:49.689280 ip-10-0-129-222 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:08:49.690882 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.690866    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:08:49.693894 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.693869    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:08:49.694301 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.694282    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:08:49.698168 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.698148    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:08:49.698324 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.698307    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:08:49.698933 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.698038    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-222.ec2.internal.18c6832c8e4cd867  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-222.ec2.internal,UID:ip-10-0-129-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-222.ec2.internal,},FirstTimestamp:2026-07-28 17:08:49.688180839 +0000 UTC m=+0.360508750,LastTimestamp:2026-07-28 17:08:49.688180839 +0000 UTC m=+0.360508750,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-222.ec2.internal,}"
Jul 28 17:08:49.699325 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.699306    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:08:49.699435 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.699325    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:08:49.699533 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.699515    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:49.699945 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.699863    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:08:49.699945 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.699864    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:08:49.699945 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.699887    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:08:49.700097 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.699976    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:08:49.700097 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.699985    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:08:49.700245 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.700228    2560 factory.go:55] Registering systemd factory
Jul 28 17:08:49.700297 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.700248    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:08:49.700482 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.700467    2560 factory.go:153] Registering CRI-O factory
Jul 28 17:08:49.700482 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.700481    2560 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:08:49.700584 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.700526    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:08:49.700584 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.700549    2560 factory.go:103] Registering Raw factory
Jul 28 17:08:49.700584 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.700565    2560 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:08:49.701020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.701006    2560 manager.go:349] Starting recovery of all containers
Jul 28 17:08:49.701950 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.701927    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:08:49.702050 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.702033    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:08:49.710582 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.710562    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jfj6g"
Jul 28 17:08:49.711737 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.711723    2560 manager.go:354] Recovery completed
Jul 28 17:08:49.713882 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.713861    2560 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 17:08:49.717153 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.717141    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:08:49.719008 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.718996    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:08:49.719056 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.719021    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:08:49.719056 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.719031    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:08:49.719315 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.719296    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jfj6g"
Jul 28 17:08:49.719547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.719536    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:08:49.719630 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.719547    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:08:49.719630 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.719562    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:08:49.719630 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.719568    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:08:49.720264 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.720245    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:08:49.720311 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.720277    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:08:49.720311 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.720289    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:08:49.721043 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.720954    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-222.ec2.internal.18c6832c9023404e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-222.ec2.internal,UID:ip-10-0-129-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-222.ec2.internal,},FirstTimestamp:2026-07-28 17:08:49.719009358 +0000 UTC m=+0.391337270,LastTimestamp:2026-07-28 17:08:49.719009358 +0000 UTC m=+0.391337270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-222.ec2.internal,}"
Jul 28 17:08:49.723031 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.723018    2560 policy_none.go:49] "None policy: Start"
Jul 28 17:08:49.723109 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.723036    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:08:49.723109 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.723049    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:08:49.724969 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.724955    2560 policy_none.go:47] "Start"
Jul 28 17:08:49.755000 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.754869    2560 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:08:49.755075 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.755019    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:08:49.755075 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.755032    2560 server.go:87] "Starting device plugin registration server"
Jul 28 17:08:49.755240 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.755229    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:08:49.755278 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.755244    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:08:49.755468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.755376    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:08:49.755468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.755452    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:08:49.755468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.755461    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:08:49.756662 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.756639    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:08:49.756787 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.756687    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:49.855778 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.855746    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:08:49.857736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.857145    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:08:49.857736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.857175    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:08:49.857736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.857187    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:08:49.857736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.857213    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-222.ec2.internal"
Jul 28 17:08:49.864707 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.864684    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:08:49.865943 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.865920    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:08:49.866031 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.865958    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:08:49.866031 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.865978    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:08:49.866031 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.866012    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:08:49.867857 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.867837    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-222.ec2.internal"
Jul 28 17:08:49.867931 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.867859    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-222.ec2.internal\": node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:49.868593 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.868575    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:08:49.928130 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.928107    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:49.966888 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.966861    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"]
Jul 28 17:08:49.967006 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.966943    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:08:49.968644 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.968629    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:08:49.968721 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.968658    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:08:49.968721 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.968670    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:08:49.970763 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.970752    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:08:49.970931 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.970919    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:49.970964 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.970947    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:08:49.971451 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.971436    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:08:49.971519 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.971457    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:08:49.971519 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.971479    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:08:49.971609 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.971558    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:08:49.971609 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.971583    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:08:49.971609 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.971592    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:08:49.973751 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.973735    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:49.973834 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.973760    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:08:49.974409 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.974392    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:08:49.974478 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.974426    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:08:49.974478 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:49.974442    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:08:49.998882 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:49.998860    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-222.ec2.internal\" not found" node="ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.001638 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.001621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e46ae8c13e8686631be2da87086a49c5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-222.ec2.internal\" (UID: \"e46ae8c13e8686631be2da87086a49c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.001737 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.001644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe59cc5e90685ef84de762ad6b19adee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"fe59cc5e90685ef84de762ad6b19adee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.001737 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.001668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe59cc5e90685ef84de762ad6b19adee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"fe59cc5e90685ef84de762ad6b19adee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.003376 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.003360    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-222.ec2.internal\" not found" node="ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.028850 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.028831    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:50.102495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.102469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe59cc5e90685ef84de762ad6b19adee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"fe59cc5e90685ef84de762ad6b19adee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.102495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.102496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe59cc5e90685ef84de762ad6b19adee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"fe59cc5e90685ef84de762ad6b19adee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.102667 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.102512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e46ae8c13e8686631be2da87086a49c5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-222.ec2.internal\" (UID: \"e46ae8c13e8686631be2da87086a49c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.102667 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.102550    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e46ae8c13e8686631be2da87086a49c5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-222.ec2.internal\" (UID: \"e46ae8c13e8686631be2da87086a49c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.102667 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.102576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe59cc5e90685ef84de762ad6b19adee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"fe59cc5e90685ef84de762ad6b19adee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.102667 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.102578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe59cc5e90685ef84de762ad6b19adee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal\" (UID: \"fe59cc5e90685ef84de762ad6b19adee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.129807 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.129780    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:50.230478 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.230389    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:50.301668 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.301640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.306320 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.306180    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.330823 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.330785    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:50.431080 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.431045    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:50.531477 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.531402    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:50.631878 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.631846    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-222.ec2.internal\" not found"
Jul 28 17:08:50.642964 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.642939    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:08:50.643117 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.643098    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:08:50.664464 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.664439    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:08:50.678435 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.678413    2560 apiserver.go:52] "Watching apiserver"
Jul 28 17:08:50.686587 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.686564    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:08:50.686944 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.686922    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-4c8k5","kube-system/global-pull-secret-syncer-ng8vd","kube-system/konnectivity-agent-txtql","openshift-cluster-node-tuning-operator/tuned-d5rhq","openshift-multus/network-metrics-daemon-n655v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47","openshift-dns/node-resolver-lgzrj","openshift-image-registry/node-ca-rgp8f","openshift-multus/multus-additional-cni-plugins-js4pf","openshift-multus/multus-kcdb4","openshift-network-diagnostics/network-check-target-jxsd6","openshift-network-operator/iptables-alerter-q7rs7"]
Jul 28 17:08:50.689412 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.689394    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.691565 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.691542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.692464 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.692443    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:08:50.692668 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.692644    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:08:50.692749 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.692678    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:08:50.692749 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.692729    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:08:50.692815 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.692761    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:08:50.692904 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.692890    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:08:50.692943 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.692907    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dq76s\""
Jul 28 17:08:50.693574 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.693550    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:08:50.693657 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.693583    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:08:50.693780 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.693768    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:50.693840 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.693824    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:08:50.694981 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.694952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wnq5m\""
Jul 28 17:08:50.697666 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.697645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.700090 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.700070    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:08:50.700090 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.700084    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.700626 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.700610    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.700917 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.700900    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:08:50.701067 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.701056    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n6fwb\""
Jul 28 17:08:50.701245 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.701231    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:08:50.702875 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.702862    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:08:50.703154 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.703136    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:08:50.703230 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.703179    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:08:50.703230 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.703203    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.703230 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.703226    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xnsvg\""
Jul 28 17:08:50.704889 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.704873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-lib-modules\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.704941 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.704901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbqft\" (UniqueName: \"kubernetes.io/projected/a6c7831c-39fd-4adc-a659-c7df60150c19-kube-api-access-vbqft\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.704941 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.704918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-kubelet\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.704941 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.704931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.704945    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-ovn\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.704961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-cni-netd\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.704993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-systemd\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-var-lib-kubelet\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:50.705134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-systemd\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-ovnkube-script-lib\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-host\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bh2\" (UniqueName: \"kubernetes.io/projected/cc7329f7-672b-4a06-81a4-d942f70a354e-kube-api-access-p4bh2\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.705293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-etc-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysctl-d\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705214    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-run\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-tuned\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6w8d\" (UniqueName: \"kubernetes.io/projected/97e85813-529b-4e8a-9c6b-96684a8dac43-kube-api-access-s6w8d\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.705293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-slash\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-cni-bin\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-env-overrides\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705355    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb247d8e-6b1f-43c4-9e25-75636655445b-ovn-node-metrics-cert\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-modprobe-d\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-sys\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705430    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6c7831c-39fd-4adc-a659-c7df60150c19-tmp\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysctl-conf\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-systemd-units\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705451    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:08:50.705527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-run-netns\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysconfig\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705540    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4zv65\""
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ndgb\" (UniqueName: \"kubernetes.io/projected/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-kube-api-access-7ndgb\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-var-lib-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705627    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-log-socket\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-ovnkube-config\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5sbs\" (UniqueName: \"kubernetes.io/projected/cb247d8e-6b1f-43c4-9e25-75636655445b-kube-api-access-g5sbs\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc7329f7-672b-4a06-81a4-d942f70a354e-tmp-dir\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e85813-529b-4e8a-9c6b-96684a8dac43-host\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e85813-529b-4e8a-9c6b-96684a8dac43-serviceca\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705775    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-node-log\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-kubernetes\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc7329f7-672b-4a06-81a4-d942f70a354e-hosts-file\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.705836    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.706068 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.705880    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:08:50.708408 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.708393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:08:50.709512 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.709499    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:08:50.709563 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.709547    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"
Jul 28 17:08:50.710549 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.710531    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:08:50.710637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.710622    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wkj9g\""
Jul 28 17:08:50.710707 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.710694    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:08:50.711033 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.711018    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.713253 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.713234    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:08:50.713436 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.713411    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dfqt4\""
Jul 28 17:08:50.713754 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.713739    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:08:50.713850 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.713835    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:08:50.713887 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.713839    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.715944 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.715924    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:08:50.716416 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.716401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.717203 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.717186    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:08:50.717293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.717215    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:08:50.717369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.717298    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:08:50.717369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.717320    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:08:50.717574 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.717493    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-87jcl\""
Jul 28 17:08:50.717574 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.717570    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:08:50.718911 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.718895    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal"]
Jul 28 17:08:50.719011 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.718955    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:50.719011 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.718993    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:08:50.719296 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.719277    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:08:50.719551 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.719535    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fcnc9\""
Jul 28 17:08:50.722389 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.722373    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal"]
Jul 28 17:08:50.722803 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.722793    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:08:50.724143 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.724113    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:03:49 +0000 UTC" deadline="2028-01-21 10:35:43.00006118 +0000 UTC"
Jul 28 17:08:50.724187 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.724142    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13001h26m52.275921167s"
Jul 28 17:08:50.753803 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.753782    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rf9p7"
Jul 28 17:08:50.762974 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.762947    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rf9p7"
Jul 28 17:08:50.801552 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.801531    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:08:50.806068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-run\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.806166 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-tuned\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.806166 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6w8d\" (UniqueName: \"kubernetes.io/projected/97e85813-529b-4e8a-9c6b-96684a8dac43-kube-api-access-s6w8d\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.806287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-etc-selinux\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.806417 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-cni-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.806417 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-run\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.806417 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-cni-binary-copy\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.806417 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806355    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-hostroot\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.806417 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806389    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:08:50.806417 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-cni-bin\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-modprobe-d\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-cni-bin\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6c7831c-39fd-4adc-a659-c7df60150c19-tmp\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-os-release\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a65b903-f10a-4ecf-abe1-4cbddd056536-konnectivity-ca\") pod \"konnectivity-agent-txtql\" (UID: \"5a65b903-f10a-4ecf-abe1-4cbddd056536\") " pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-modprobe-d\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-cnibin\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-multus-certs\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swn7b\" (UniqueName: \"kubernetes.io/projected/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-kube-api-access-swn7b\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-var-lib-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-log-socket\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.806691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806679    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-var-lib-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc7329f7-672b-4a06-81a4-d942f70a354e-tmp-dir\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e85813-529b-4e8a-9c6b-96684a8dac43-host\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-log-socket\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-socket-dir-parent\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-cni-bin\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e85813-529b-4e8a-9c6b-96684a8dac43-host\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-kubelet\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-node-log\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-kubernetes\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-node-log\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-kubernetes\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc7329f7-672b-4a06-81a4-d942f70a354e-hosts-file\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc7329f7-672b-4a06-81a4-d942f70a354e-hosts-file\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.807222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc7329f7-672b-4a06-81a4-d942f70a354e-tmp-dir\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.806984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-registration-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh64n\" (UniqueName: \"kubernetes.io/projected/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-kube-api-access-zh64n\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a65b903-f10a-4ecf-abe1-4cbddd056536-agent-certs\") pod \"konnectivity-agent-txtql\" (UID: \"5a65b903-f10a-4ecf-abe1-4cbddd056536\") " pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-lib-modules\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-kubelet\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-systemd\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-kubelet\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-host-slash\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-systemd\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-lib-modules\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-ovnkube-script-lib\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807183    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b207869a-8b53-42dd-a318-2645c2d55a3d-kubelet-config\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807250    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-cnibin\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-netns\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-cni-binary-copy\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.808018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-etc-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-iptables-alerter-script\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-etc-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-slash\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-slash\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-env-overrides\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb247d8e-6b1f-43c4-9e25-75636655445b-ovn-node-metrics-cert\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-sys\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-k8s-cni-cncf-io\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-sys\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-cni-multus\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk8zn\" (UniqueName: \"kubernetes.io/projected/523beaef-da25-4143-890d-e6eab2507a01-kube-api-access-bk8zn\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysctl-conf\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-env-overrides\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-systemd-units\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-ovnkube-script-lib\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-run-netns\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.808830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysconfig\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-systemd-units\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndgb\" (UniqueName: \"kubernetes.io/projected/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-kube-api-access-7ndgb\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b207869a-8b53-42dd-a318-2645c2d55a3d-dbus\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807864    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysconfig\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807869    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-device-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-run-netns\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-ovnkube-config\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5sbs\" (UniqueName: \"kubernetes.io/projected/cb247d8e-6b1f-43c4-9e25-75636655445b-kube-api-access-g5sbs\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.807959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e85813-529b-4e8a-9c6b-96684a8dac43-serviceca\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysctl-conf\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-etc-kubernetes\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-conf-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-daemon-config\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbqft\" (UniqueName: \"kubernetes.io/projected/a6c7831c-39fd-4adc-a659-c7df60150c19-kube-api-access-vbqft\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-socket-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.809547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-sys-fs\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqnr7\" (UniqueName: \"kubernetes.io/projected/3b1158c2-54a3-43a8-8487-dee5c1b6b737-kube-api-access-jqnr7\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-ovn\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-cni-netd\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-var-lib-kubelet\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bh2\" (UniqueName: \"kubernetes.io/projected/cc7329f7-672b-4a06-81a4-d942f70a354e-kube-api-access-p4bh2\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-os-release\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-systemd\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-host\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-system-cni-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-system-cni-dir\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-ovn\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-openvswitch\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-var-lib-kubelet\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-cni-netd\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e85813-529b-4e8a-9c6b-96684a8dac43-serviceca\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808876    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-run-systemd\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb247d8e-6b1f-43c4-9e25-75636655445b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.808921    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-host\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.808969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.809025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysctl-d\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.809123    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs podName:3aaf58b5-67e4-4a7f-b5fe-472d3968101e nodeName:}" failed. No retries permitted until 2026-07-28 17:08:51.309101093 +0000 UTC m=+1.981429021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs") pod "network-metrics-daemon-n655v" (UID: "3aaf58b5-67e4-4a7f-b5fe-472d3968101e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.809235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-sysctl-d\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.809835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6c7831c-39fd-4adc-a659-c7df60150c19-etc-tuned\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.810670 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.809990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6c7831c-39fd-4adc-a659-c7df60150c19-tmp\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.811072 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.810769    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb247d8e-6b1f-43c4-9e25-75636655445b-ovn-node-metrics-cert\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.811130 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.811119    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb247d8e-6b1f-43c4-9e25-75636655445b-ovnkube-config\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.814503 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.814478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6w8d\" (UniqueName: \"kubernetes.io/projected/97e85813-529b-4e8a-9c6b-96684a8dac43-kube-api-access-s6w8d\") pod \"node-ca-rgp8f\" (UID: \"97e85813-529b-4e8a-9c6b-96684a8dac43\") " pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:50.817386 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.817365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbqft\" (UniqueName: \"kubernetes.io/projected/a6c7831c-39fd-4adc-a659-c7df60150c19-kube-api-access-vbqft\") pod \"tuned-d5rhq\" (UID: \"a6c7831c-39fd-4adc-a659-c7df60150c19\") " pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:50.817502 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.817478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndgb\" (UniqueName: \"kubernetes.io/projected/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-kube-api-access-7ndgb\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:50.818455 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.818438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5sbs\" (UniqueName: \"kubernetes.io/projected/cb247d8e-6b1f-43c4-9e25-75636655445b-kube-api-access-g5sbs\") pod \"ovnkube-node-4c8k5\" (UID: \"cb247d8e-6b1f-43c4-9e25-75636655445b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:50.819308 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.819293    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bh2\" (UniqueName: \"kubernetes.io/projected/cc7329f7-672b-4a06-81a4-d942f70a354e-kube-api-access-p4bh2\") pod \"node-resolver-lgzrj\" (UID: \"cc7329f7-672b-4a06-81a4-d942f70a354e\") " pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:50.878703 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:50.878668    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe59cc5e90685ef84de762ad6b19adee.slice/crio-ba342c56fe5a036b9e329c9410eacb58c2780ee4f8343f75a4d5f157aa19d7f8 WatchSource:0}: Error finding container ba342c56fe5a036b9e329c9410eacb58c2780ee4f8343f75a4d5f157aa19d7f8: Status 404 returned error can't find the container with id ba342c56fe5a036b9e329c9410eacb58c2780ee4f8343f75a4d5f157aa19d7f8
Jul 28 17:08:50.879043 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:50.879023    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46ae8c13e8686631be2da87086a49c5.slice/crio-ac66b2b949037a782e5ed5c7f6d1dd55897769f6e03fbcfd318fdd466528aa27 WatchSource:0}: Error finding container ac66b2b949037a782e5ed5c7f6d1dd55897769f6e03fbcfd318fdd466528aa27: Status 404 returned error can't find the container with id ac66b2b949037a782e5ed5c7f6d1dd55897769f6e03fbcfd318fdd466528aa27
Jul 28 17:08:50.885287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.885266    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:08:50.910128 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-os-release\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.910236 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-system-cni-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910236 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-system-cni-dir\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.910236 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.910236 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-system-cni-dir\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-os-release\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-etc-selinux\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-cni-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-cni-binary-copy\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-system-cni-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-hostroot\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910432 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-os-release\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a65b903-f10a-4ecf-abe1-4cbddd056536-konnectivity-ca\") pod \"konnectivity-agent-txtql\" (UID: \"5a65b903-f10a-4ecf-abe1-4cbddd056536\") " pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-cnibin\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-etc-selinux\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910492    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-cni-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-hostroot\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/523beaef-da25-4143-890d-e6eab2507a01-cnibin\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-multus-certs\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-os-release\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910553    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-multus-certs\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swn7b\" (UniqueName: \"kubernetes.io/projected/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-kube-api-access-swn7b\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-socket-dir-parent\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-cni-bin\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-kubelet\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-cni-bin\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-registration-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.910787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910687    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-socket-dir-parent\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh64n\" (UniqueName: \"kubernetes.io/projected/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-kube-api-access-zh64n\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-kubelet\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-registration-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a65b903-f10a-4ecf-abe1-4cbddd056536-agent-certs\") pod \"konnectivity-agent-txtql\" (UID: \"5a65b903-f10a-4ecf-abe1-4cbddd056536\") " pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-host-slash\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.910848    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.910911    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret podName:b207869a-8b53-42dd-a318-2645c2d55a3d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:51.41088019 +0000 UTC m=+2.083208102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret") pod "global-pull-secret-syncer-ng8vd" (UID: "b207869a-8b53-42dd-a318-2645c2d55a3d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-host-slash\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b207869a-8b53-42dd-a318-2645c2d55a3d-kubelet-config\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-cni-binary-copy\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-cnibin\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-netns\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.910997    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a65b903-f10a-4ecf-abe1-4cbddd056536-konnectivity-ca\") pod \"konnectivity-agent-txtql\" (UID: \"5a65b903-f10a-4ecf-abe1-4cbddd056536\") " pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-cni-binary-copy\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.911637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b207869a-8b53-42dd-a318-2645c2d55a3d-kubelet-config\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-iptables-alerter-script\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911072    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-netns\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-cnibin\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-k8s-cni-cncf-io\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-cni-multus\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk8zn\" (UniqueName: \"kubernetes.io/projected/523beaef-da25-4143-890d-e6eab2507a01-kube-api-access-bk8zn\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-run-k8s-cni-cncf-io\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-host-var-lib-cni-multus\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911193    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b207869a-8b53-42dd-a318-2645c2d55a3d-dbus\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-device-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-etc-kubernetes\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-conf-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-daemon-config\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911305    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-device-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-socket-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.912195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-sys-fs\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b207869a-8b53-42dd-a318-2645c2d55a3d-dbus\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqnr7\" (UniqueName: \"kubernetes.io/projected/3b1158c2-54a3-43a8-8487-dee5c1b6b737-kube-api-access-jqnr7\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-conf-dir\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911477    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-etc-kubernetes\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911560    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-cni-binary-copy\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-sys-fs\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b1158c2-54a3-43a8-8487-dee5c1b6b737-socket-dir\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-iptables-alerter-script\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.911974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/523beaef-da25-4143-890d-e6eab2507a01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.912752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.912067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-multus-daemon-config\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.913182 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.913167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a65b903-f10a-4ecf-abe1-4cbddd056536-agent-certs\") pod \"konnectivity-agent-txtql\" (UID: \"5a65b903-f10a-4ecf-abe1-4cbddd056536\") " pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:08:50.921483 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.921467    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:08:50.921550 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.921487    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:08:50.921550 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.921496    2560 projected.go:196] Error preparing data for projected volume kube-api-access-qg5qr for pod openshift-network-diagnostics/network-check-target-jxsd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:50.921550 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:50.921539    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr podName:1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:51.42152699 +0000 UTC m=+2.093854910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qg5qr" (UniqueName: "kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr") pod "network-check-target-jxsd6" (UID: "1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:50.924827 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.924809    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swn7b\" (UniqueName: \"kubernetes.io/projected/1b90386f-7ec6-4c92-b9c0-aa78ec25bffb-kube-api-access-swn7b\") pod \"multus-kcdb4\" (UID: \"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb\") " pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:50.925027 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.925006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh64n\" (UniqueName: \"kubernetes.io/projected/7e62d46f-cfd7-429d-8a96-cd81ba37fe40-kube-api-access-zh64n\") pod \"iptables-alerter-q7rs7\" (UID: \"7e62d46f-cfd7-429d-8a96-cd81ba37fe40\") " pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:50.925488 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.925470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqnr7\" (UniqueName: \"kubernetes.io/projected/3b1158c2-54a3-43a8-8487-dee5c1b6b737-kube-api-access-jqnr7\") pod \"aws-ebs-csi-driver-node-47t47\" (UID: \"3b1158c2-54a3-43a8-8487-dee5c1b6b737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:50.926333 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.926316    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk8zn\" (UniqueName: \"kubernetes.io/projected/523beaef-da25-4143-890d-e6eab2507a01-kube-api-access-bk8zn\") pod \"multus-additional-cni-plugins-js4pf\" (UID: \"523beaef-da25-4143-890d-e6eab2507a01\") " pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:50.981988 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:50.981963    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:08:51.019517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.019497    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:08:51.028577 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.028555    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb247d8e_6b1f_43c4_9e25_75636655445b.slice/crio-381838eab3bd9e2b6442c2a720df41a0ddc01604fc375365c4111ab577aebde4 WatchSource:0}: Error finding container 381838eab3bd9e2b6442c2a720df41a0ddc01604fc375365c4111ab577aebde4: Status 404 returned error can't find the container with id 381838eab3bd9e2b6442c2a720df41a0ddc01604fc375365c4111ab577aebde4
Jul 28 17:08:51.031577 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.031560    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d5rhq"
Jul 28 17:08:51.039456 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.039437    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c7831c_39fd_4adc_a659_c7df60150c19.slice/crio-923e8da806ca514abd80c96d817fc3f38ffbabfe91788fd327a5e7e491b65792 WatchSource:0}: Error finding container 923e8da806ca514abd80c96d817fc3f38ffbabfe91788fd327a5e7e491b65792: Status 404 returned error can't find the container with id 923e8da806ca514abd80c96d817fc3f38ffbabfe91788fd327a5e7e491b65792
Jul 28 17:08:51.049107 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.049069    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lgzrj"
Jul 28 17:08:51.056912 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.056896    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7329f7_672b_4a06_81a4_d942f70a354e.slice/crio-b2d4b7ed4621a942babde41092b028d95b2093cf215fdaa297772989dde00048 WatchSource:0}: Error finding container b2d4b7ed4621a942babde41092b028d95b2093cf215fdaa297772989dde00048: Status 404 returned error can't find the container with id b2d4b7ed4621a942babde41092b028d95b2093cf215fdaa297772989dde00048
Jul 28 17:08:51.068353 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.068328    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rgp8f"
Jul 28 17:08:51.075955 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.075937    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e85813_529b_4e8a_9c6b_96684a8dac43.slice/crio-a99b4519318cf1ca454cc7f292e67d547c560a90f678e7224c3b121829b4dfa9 WatchSource:0}: Error finding container a99b4519318cf1ca454cc7f292e67d547c560a90f678e7224c3b121829b4dfa9: Status 404 returned error can't find the container with id a99b4519318cf1ca454cc7f292e67d547c560a90f678e7224c3b121829b4dfa9
Jul 28 17:08:51.090880 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.090858    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q7rs7"
Jul 28 17:08:51.096558 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.096538    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:08:51.098309 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.098270    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e62d46f_cfd7_429d_8a96_cd81ba37fe40.slice/crio-0c1559ef24f813d824a0d1d9a9fc9e3150d8c72dbf633d258da01e567b890586 WatchSource:0}: Error finding container 0c1559ef24f813d824a0d1d9a9fc9e3150d8c72dbf633d258da01e567b890586: Status 404 returned error can't find the container with id 0c1559ef24f813d824a0d1d9a9fc9e3150d8c72dbf633d258da01e567b890586
Jul 28 17:08:51.104307 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.104288    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a65b903_f10a_4ecf_abe1_4cbddd056536.slice/crio-5b8289bb239b6d7a53520f758df598b8d86c7120090d41fff2ed357d3358f0bf WatchSource:0}: Error finding container 5b8289bb239b6d7a53520f758df598b8d86c7120090d41fff2ed357d3358f0bf: Status 404 returned error can't find the container with id 5b8289bb239b6d7a53520f758df598b8d86c7120090d41fff2ed357d3358f0bf
Jul 28 17:08:51.129873 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.129853    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:08:51.131742 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.131726    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47"
Jul 28 17:08:51.140555 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.140538    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1158c2_54a3_43a8_8487_dee5c1b6b737.slice/crio-13f410065bcad94cebd06f86b545929a728a76fb5996299b44bc6299378224be WatchSource:0}: Error finding container 13f410065bcad94cebd06f86b545929a728a76fb5996299b44bc6299378224be: Status 404 returned error can't find the container with id 13f410065bcad94cebd06f86b545929a728a76fb5996299b44bc6299378224be
Jul 28 17:08:51.147314 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.147299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-js4pf"
Jul 28 17:08:51.152041 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.152027    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kcdb4"
Jul 28 17:08:51.155216 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.155195    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523beaef_da25_4143_890d_e6eab2507a01.slice/crio-3f144bc5768b156f80e6182210ff6a86a8b491fa9916191fbcbddeafea1e8e72 WatchSource:0}: Error finding container 3f144bc5768b156f80e6182210ff6a86a8b491fa9916191fbcbddeafea1e8e72: Status 404 returned error can't find the container with id 3f144bc5768b156f80e6182210ff6a86a8b491fa9916191fbcbddeafea1e8e72
Jul 28 17:08:51.160420 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:08:51.160399    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b90386f_7ec6_4c92_b9c0_aa78ec25bffb.slice/crio-16984608f838feafc6d551bfdc9dbdef2e53d307db51701da695de299076c049 WatchSource:0}: Error finding container 16984608f838feafc6d551bfdc9dbdef2e53d307db51701da695de299076c049: Status 404 returned error can't find the container with id 16984608f838feafc6d551bfdc9dbdef2e53d307db51701da695de299076c049
Jul 28 17:08:51.313787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.313706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:51.313908 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.313853    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:51.313958 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.313916    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs podName:3aaf58b5-67e4-4a7f-b5fe-472d3968101e nodeName:}" failed. No retries permitted until 2026-07-28 17:08:52.313901351 +0000 UTC m=+2.986229250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs") pod "network-metrics-daemon-n655v" (UID: "3aaf58b5-67e4-4a7f-b5fe-472d3968101e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:51.414523 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.414489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:51.414707 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.414622    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:51.414707 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.414680    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret podName:b207869a-8b53-42dd-a318-2645c2d55a3d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:52.414663008 +0000 UTC m=+3.086990912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret") pod "global-pull-secret-syncer-ng8vd" (UID: "b207869a-8b53-42dd-a318-2645c2d55a3d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:51.515406 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.515369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:51.515593 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.515563    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:08:51.515593 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.515581    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:08:51.515593 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.515593    2560 projected.go:196] Error preparing data for projected volume kube-api-access-qg5qr for pod openshift-network-diagnostics/network-check-target-jxsd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:51.515748 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.515652    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr podName:1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:52.51563307 +0000 UTC m=+3.187960982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qg5qr" (UniqueName: "kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr") pod "network-check-target-jxsd6" (UID: "1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:51.587272 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.587198    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:08:51.766635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.766571    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:03:50 +0000 UTC" deadline="2028-01-18 16:49:26.838510258 +0000 UTC"
Jul 28 17:08:51.766635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.766631    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12935h40m35.071883196s"
Jul 28 17:08:51.868629 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.868559    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:51.868785 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:51.868690    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:08:51.885745 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.885611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-txtql" event={"ID":"5a65b903-f10a-4ecf-abe1-4cbddd056536","Type":"ContainerStarted","Data":"5b8289bb239b6d7a53520f758df598b8d86c7120090d41fff2ed357d3358f0bf"}
Jul 28 17:08:51.895152 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.895112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rgp8f" event={"ID":"97e85813-529b-4e8a-9c6b-96684a8dac43","Type":"ContainerStarted","Data":"a99b4519318cf1ca454cc7f292e67d547c560a90f678e7224c3b121829b4dfa9"}
Jul 28 17:08:51.904491 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.904165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lgzrj" event={"ID":"cc7329f7-672b-4a06-81a4-d942f70a354e","Type":"ContainerStarted","Data":"b2d4b7ed4621a942babde41092b028d95b2093cf215fdaa297772989dde00048"}
Jul 28 17:08:51.915475 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.915444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d5rhq" event={"ID":"a6c7831c-39fd-4adc-a659-c7df60150c19","Type":"ContainerStarted","Data":"923e8da806ca514abd80c96d817fc3f38ffbabfe91788fd327a5e7e491b65792"}
Jul 28 17:08:51.927426 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.927400    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"381838eab3bd9e2b6442c2a720df41a0ddc01604fc375365c4111ab577aebde4"}
Jul 28 17:08:51.937519 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.937463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal" event={"ID":"e46ae8c13e8686631be2da87086a49c5","Type":"ContainerStarted","Data":"ac66b2b949037a782e5ed5c7f6d1dd55897769f6e03fbcfd318fdd466528aa27"}
Jul 28 17:08:51.947427 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.947396    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal" event={"ID":"fe59cc5e90685ef84de762ad6b19adee","Type":"ContainerStarted","Data":"ba342c56fe5a036b9e329c9410eacb58c2780ee4f8343f75a4d5f157aa19d7f8"}
Jul 28 17:08:51.974359 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.974317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kcdb4" event={"ID":"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb","Type":"ContainerStarted","Data":"16984608f838feafc6d551bfdc9dbdef2e53d307db51701da695de299076c049"}
Jul 28 17:08:51.992465 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:51.992392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4pf" event={"ID":"523beaef-da25-4143-890d-e6eab2507a01","Type":"ContainerStarted","Data":"3f144bc5768b156f80e6182210ff6a86a8b491fa9916191fbcbddeafea1e8e72"}
Jul 28 17:08:52.015927 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.015894    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47" event={"ID":"3b1158c2-54a3-43a8-8487-dee5c1b6b737","Type":"ContainerStarted","Data":"13f410065bcad94cebd06f86b545929a728a76fb5996299b44bc6299378224be"}
Jul 28 17:08:52.031708 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.029867    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q7rs7" event={"ID":"7e62d46f-cfd7-429d-8a96-cd81ba37fe40","Type":"ContainerStarted","Data":"0c1559ef24f813d824a0d1d9a9fc9e3150d8c72dbf633d258da01e567b890586"}
Jul 28 17:08:52.323334 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.322699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:52.323334 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.322882    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:52.323334 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.322944    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs podName:3aaf58b5-67e4-4a7f-b5fe-472d3968101e nodeName:}" failed. No retries permitted until 2026-07-28 17:08:54.322925896 +0000 UTC m=+4.995253802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs") pod "network-metrics-daemon-n655v" (UID: "3aaf58b5-67e4-4a7f-b5fe-472d3968101e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:52.423146 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.423083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:52.423304 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.423222    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:52.423304 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.423280    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret podName:b207869a-8b53-42dd-a318-2645c2d55a3d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:54.4232612 +0000 UTC m=+5.095589103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret") pod "global-pull-secret-syncer-ng8vd" (UID: "b207869a-8b53-42dd-a318-2645c2d55a3d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:52.524247 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.524209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:52.524456 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.524413    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:08:52.524456 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.524437    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:08:52.524456 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.524451    2560 projected.go:196] Error preparing data for projected volume kube-api-access-qg5qr for pod openshift-network-diagnostics/network-check-target-jxsd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:52.524627 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.524512    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr podName:1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:54.524492863 +0000 UTC m=+5.196820768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qg5qr" (UniqueName: "kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr") pod "network-check-target-jxsd6" (UID: "1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:52.767553 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.767454    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:03:50 +0000 UTC" deadline="2028-01-08 21:13:54.214054047 +0000 UTC"
Jul 28 17:08:52.767553 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.767497    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12700h5m1.446561349s"
Jul 28 17:08:52.866215 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.866187    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:52.866411 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.866299    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:08:52.866669 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:52.866641    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:52.866765 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:52.866725    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:08:53.868476 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:53.868388    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:53.869034 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:53.868537    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:08:54.343555 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:54.343471    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:54.343720 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.343668    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:54.343800 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.343738    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs podName:3aaf58b5-67e4-4a7f-b5fe-472d3968101e nodeName:}" failed. No retries permitted until 2026-07-28 17:08:58.34371852 +0000 UTC m=+9.016046424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs") pod "network-metrics-daemon-n655v" (UID: "3aaf58b5-67e4-4a7f-b5fe-472d3968101e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:54.445168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:54.444604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:54.445168 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.444769    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:54.445168 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.444832    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret podName:b207869a-8b53-42dd-a318-2645c2d55a3d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:58.444813902 +0000 UTC m=+9.117141807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret") pod "global-pull-secret-syncer-ng8vd" (UID: "b207869a-8b53-42dd-a318-2645c2d55a3d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:54.545400 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:54.545362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:54.545580 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.545528    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:08:54.545580 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.545550    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:08:54.545580 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.545564    2560 projected.go:196] Error preparing data for projected volume kube-api-access-qg5qr for pod openshift-network-diagnostics/network-check-target-jxsd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:54.545727 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.545622    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr podName:1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:58.545603688 +0000 UTC m=+9.217931607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qg5qr" (UniqueName: "kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr") pod "network-check-target-jxsd6" (UID: "1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:54.867070 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:54.866485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:54.867070 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:54.866533    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:54.867070 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.866628    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:08:54.867070 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:54.867028    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:08:55.869303 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:55.868777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:55.869303 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:55.868909    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:08:56.866492 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:56.866454    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:56.866686 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:56.866590    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:08:56.866686 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:56.866455    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:56.866822 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:56.866718    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:08:57.866808 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:57.866726    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:57.867234 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:57.866888    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:08:58.380519 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:58.380483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:58.380717 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.380614    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:58.380717 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.380684    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs podName:3aaf58b5-67e4-4a7f-b5fe-472d3968101e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:06.380664773 +0000 UTC m=+17.052992672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs") pod "network-metrics-daemon-n655v" (UID: "3aaf58b5-67e4-4a7f-b5fe-472d3968101e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:08:58.482105 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:58.481534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:58.482105 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.481743    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:58.482105 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.481812    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret podName:b207869a-8b53-42dd-a318-2645c2d55a3d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:06.481794363 +0000 UTC m=+17.154122266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret") pod "global-pull-secret-syncer-ng8vd" (UID: "b207869a-8b53-42dd-a318-2645c2d55a3d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:08:58.582484 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:58.582441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:58.582669 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.582618    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:08:58.582669 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.582639    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:08:58.582669 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.582648    2560 projected.go:196] Error preparing data for projected volume kube-api-access-qg5qr for pod openshift-network-diagnostics/network-check-target-jxsd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:58.582831 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.582691    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr podName:1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:06.582678694 +0000 UTC m=+17.255006606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qg5qr" (UniqueName: "kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr") pod "network-check-target-jxsd6" (UID: "1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:08:58.866632 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:58.866506    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:08:58.866787 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.866655    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:08:58.866930 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:58.866879    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:08:58.867284 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:58.867002    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:08:59.868047 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:08:59.868004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:08:59.868521 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:08:59.868213    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:00.866286 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:00.866247    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:00.866286 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:00.866279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:00.866521 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:00.866383    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:00.866521 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:00.866492    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:01.866545 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:01.866515    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:01.866987 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:01.866681    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:02.866938 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:02.866901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:02.867374 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:02.866901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:02.867374 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:02.867031    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:02.867374 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:02.867085    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:03.866598 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:03.866561    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:03.866774 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:03.866697    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:04.866942 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:04.866914    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:04.867403 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:04.867029    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:04.867403 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:04.867083    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:04.867403 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:04.867186    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:05.866726 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:05.866693    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:05.866893 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:05.866832    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:06.445850 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:06.445804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:06.446298 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.445960    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:06.446298 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.446035    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs podName:3aaf58b5-67e4-4a7f-b5fe-472d3968101e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.446009624 +0000 UTC m=+33.118337524 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs") pod "network-metrics-daemon-n655v" (UID: "3aaf58b5-67e4-4a7f-b5fe-472d3968101e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:06.546527 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:06.546494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:06.546725 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.546684    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:06.546791 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.546761    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret podName:b207869a-8b53-42dd-a318-2645c2d55a3d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.546738988 +0000 UTC m=+33.219066890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret") pod "global-pull-secret-syncer-ng8vd" (UID: "b207869a-8b53-42dd-a318-2645c2d55a3d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:06.647811 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:06.647769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:06.648004 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.647932    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:06.648004 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.647961    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:06.648004 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.647975    2560 projected.go:196] Error preparing data for projected volume kube-api-access-qg5qr for pod openshift-network-diagnostics/network-check-target-jxsd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:06.648145 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.648038    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr podName:1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.648016368 +0000 UTC m=+33.320344266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qg5qr" (UniqueName: "kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr") pod "network-check-target-jxsd6" (UID: "1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:06.866166 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:06.866134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:06.866353 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.866235    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:06.866353 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:06.866265    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:06.866353 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:06.866321    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:07.866494 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:07.866458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:07.866885 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:07.866621    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:08.866734 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:08.866700    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:08.867208 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:08.866700    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:08.867208 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:08.866832    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:08.867208 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:08.866900    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:09.867647 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:09.867609    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:09.868119 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:09.867744    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:10.867241 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:10.867095    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:10.867357 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:10.867100    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:10.867357 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:10.867310    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:10.867475 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:10.867426    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:11.084514 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:11.084480    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d5rhq" event={"ID":"a6c7831c-39fd-4adc-a659-c7df60150c19","Type":"ContainerStarted","Data":"f35d1037e2e44d7ba4dbd40112c038e16bed0d9dc869a90813363edf12ea5a9c"}
Jul 28 17:09:11.089204 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:11.089129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal" event={"ID":"e46ae8c13e8686631be2da87086a49c5","Type":"ContainerStarted","Data":"68b287ea5fac4470d69f7ab6226c4b757f40564cf6f74b82b3c4e32732d97977"}
Jul 28 17:09:11.116881 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:11.116825    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d5rhq" podStartSLOduration=2.408471696 podStartE2EDuration="22.116808026s" totalImagesPullingTime="19.70833633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:11.115676429 +0000 UTC m=+21.788004356" watchObservedRunningTime="2026-07-28 17:09:11.116808026 +0000 UTC m=+21.789135947"
Jul 28 17:09:11.137493 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:11.137459    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-222.ec2.internal" podStartSLOduration=21.137442795 podStartE2EDuration="21.137442795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:11.137297676 +0000 UTC m=+21.809625595" watchObservedRunningTime="2026-07-28 17:09:11.137442795 +0000 UTC m=+21.809770716"
Jul 28 17:09:11.867403 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:11.867220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:11.867651 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:11.867499    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:12.092130 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.092097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47" event={"ID":"3b1158c2-54a3-43a8-8487-dee5c1b6b737","Type":"ContainerStarted","Data":"846102a84e3ede68e4c6360b7b0e6e5562572e54e7a95805fb2991d844ee5ddd"}
Jul 28 17:09:12.093382 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.093363    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q7rs7" event={"ID":"7e62d46f-cfd7-429d-8a96-cd81ba37fe40","Type":"ContainerStarted","Data":"b083794fe5148d1133f8b6a3a52fe647627facb657d1b36c2b9a370eec76450b"}
Jul 28 17:09:12.094719 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.094699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-txtql" event={"ID":"5a65b903-f10a-4ecf-abe1-4cbddd056536","Type":"ContainerStarted","Data":"40bdd455ca9f3d162c5307607aa7afb04b25809c8b8c0f8020774f06ebac861d"}
Jul 28 17:09:12.095964 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.095945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rgp8f" event={"ID":"97e85813-529b-4e8a-9c6b-96684a8dac43","Type":"ContainerStarted","Data":"f684bc227504283413216d5e3c90ede71142c917b24eeafe534867c67f6e9f26"}
Jul 28 17:09:12.097285 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.097263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lgzrj" event={"ID":"cc7329f7-672b-4a06-81a4-d942f70a354e","Type":"ContainerStarted","Data":"849240f86b6dc67058a9a4ecabc29707300dda085ccc2b3970245deb946f6704"}
Jul 28 17:09:12.099742 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.099725    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:09:12.100041 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.100023    2560 generic.go:356] "Generic (PLEG): container finished" podID="cb247d8e-6b1f-43c4-9e25-75636655445b" containerID="ebe72ad3ca799695ca6823e52597abfc4087af53b456160f1c924a7c36d9b98f" exitCode=1
Jul 28 17:09:12.100113 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.100090    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"02c4799a33998da58b3cf464f8da5fa65ad6757907a889d15209119c9a1847cb"}
Jul 28 17:09:12.100171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.100157    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"78cd7c96d84145a24fc2ffdfb3e7b2eaa3851c1db4abc4ceebebb472140df7b6"}
Jul 28 17:09:12.100218 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.100207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"2df8359b7dfd479521f07f04c7880077fae1029a736c8874314e83ed24381a15"}
Jul 28 17:09:12.100269 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.100258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"bde480807137dfe2a4ada512682d9841bc19f358e04e9c96ccb519915e1b56ed"}
Jul 28 17:09:12.100318 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.100306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"912bfa182e8795434128b29ed8910b7536cb398558420c8dee891f374f1d0d00"}
Jul 28 17:09:12.100390 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.100366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerDied","Data":"ebe72ad3ca799695ca6823e52597abfc4087af53b456160f1c924a7c36d9b98f"}
Jul 28 17:09:12.101357 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.101325    2560 generic.go:356] "Generic (PLEG): container finished" podID="fe59cc5e90685ef84de762ad6b19adee" containerID="d1534e9f285a7cba367804600a6cb60e6e1c22d53b268e7e0d202b95241438d3" exitCode=0
Jul 28 17:09:12.101442 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.101401    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal" event={"ID":"fe59cc5e90685ef84de762ad6b19adee","Type":"ContainerDied","Data":"d1534e9f285a7cba367804600a6cb60e6e1c22d53b268e7e0d202b95241438d3"}
Jul 28 17:09:12.102721 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.102701    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kcdb4" event={"ID":"1b90386f-7ec6-4c92-b9c0-aa78ec25bffb","Type":"ContainerStarted","Data":"31a5719fcc2f6a07a9e2a81c6b9b4ceed213b127a5d296bcbda3f2e6bbbb6991"}
Jul 28 17:09:12.104088 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.104066    2560 generic.go:356] "Generic (PLEG): container finished" podID="523beaef-da25-4143-890d-e6eab2507a01" containerID="0a98d47293e7d5adcd159908740da1d2ce43b73d455b3efdcbc0cd1683b57793" exitCode=0
Jul 28 17:09:12.104169 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.104139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4pf" event={"ID":"523beaef-da25-4143-890d-e6eab2507a01","Type":"ContainerDied","Data":"0a98d47293e7d5adcd159908740da1d2ce43b73d455b3efdcbc0cd1683b57793"}
Jul 28 17:09:12.117009 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.116975    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q7rs7" podStartSLOduration=3.404270212 podStartE2EDuration="23.116964748s" totalImagesPullingTime="19.712694536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:12.116881983 +0000 UTC m=+22.789209904" watchObservedRunningTime="2026-07-28 17:09:12.116964748 +0000 UTC m=+22.789292667"
Jul 28 17:09:12.135005 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.134966    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lgzrj" podStartSLOduration=3.374587698 podStartE2EDuration="23.134950844s" totalImagesPullingTime="19.760363146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:12.134304117 +0000 UTC m=+22.806632036" watchObservedRunningTime="2026-07-28 17:09:12.134950844 +0000 UTC m=+22.807278765"
Jul 28 17:09:12.162648 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.162203    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-txtql" podStartSLOduration=2.518704113 podStartE2EDuration="22.162185982s" totalImagesPullingTime="19.643481869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:12.162118752 +0000 UTC m=+22.834446671" watchObservedRunningTime="2026-07-28 17:09:12.162185982 +0000 UTC m=+22.834513903"
Jul 28 17:09:12.199321 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.199287    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rgp8f" podStartSLOduration=3.462599393 podStartE2EDuration="23.199268062s" totalImagesPullingTime="19.736668669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:12.199157103 +0000 UTC m=+22.871485038" watchObservedRunningTime="2026-07-28 17:09:12.199268062 +0000 UTC m=+22.871595985"
Jul 28 17:09:12.265626 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.265581    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kcdb4" podStartSLOduration=2.363798942 podStartE2EDuration="22.265556778s" totalImagesPullingTime="19.901757836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:12.26516673 +0000 UTC m=+22.937494652" watchObservedRunningTime="2026-07-28 17:09:12.265556778 +0000 UTC m=+22.937884698"
Jul 28 17:09:12.802855 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.802830    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:09:12.866536 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.866421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:12.866725 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:12.866543    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:12.866725 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:12.866602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:12.866840 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:12.866732    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:13.108205 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:13.108168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47" event={"ID":"3b1158c2-54a3-43a8-8487-dee5c1b6b737","Type":"ContainerStarted","Data":"1389a8adcd56bc14658fc69e84903b1a3cc75b3c0728c68b40435fb07343c67e"}
Jul 28 17:09:13.110051 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:13.109982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal" event={"ID":"fe59cc5e90685ef84de762ad6b19adee","Type":"ContainerStarted","Data":"87aa672f7b125ad4993f59c6ca7a1ffab535f7cd6d47946f8e2f5d5dc3c4820b"}
Jul 28 17:09:13.771134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:13.771038    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:09:12.8028506Z","UUID":"87722ab6-88e1-449d-aedd-bb8f2dccc401","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:09:13.774279 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:13.774247    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:09:13.774279 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:13.774284    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:09:13.866588 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:13.866561    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:13.866770 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:13.866746    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:14.115937 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:14.115906    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:09:14.116396 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:14.116369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"88f9c5120680c76886da706bf39613251a12e15df00f731f20f01b0067e64867"}
Jul 28 17:09:14.368745 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:14.368717    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:09:14.369446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:14.369424    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:09:14.394882 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:14.394839    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-222.ec2.internal" podStartSLOduration=24.394823375 podStartE2EDuration="24.394823375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:13.14608317 +0000 UTC m=+23.818411086" watchObservedRunningTime="2026-07-28 17:09:14.394823375 +0000 UTC m=+25.067151295"
Jul 28 17:09:14.866553 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:14.866510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:14.866702 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:14.866643    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:14.866765 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:14.866701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:14.866834 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:14.866808    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:15.121551 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:15.121468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47" event={"ID":"3b1158c2-54a3-43a8-8487-dee5c1b6b737","Type":"ContainerStarted","Data":"3a51dc8afbaeffe33d814be51cc059d3563e357889450278d6503f05d59be08a"}
Jul 28 17:09:15.121997 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:15.121648    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:09:15.122335 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:15.122315    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-txtql"
Jul 28 17:09:15.146692 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:15.146651    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47t47" podStartSLOduration=2.990195604 podStartE2EDuration="25.146635262s" totalImagesPullingTime="22.156439658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:15.146025487 +0000 UTC m=+25.818353410" watchObservedRunningTime="2026-07-28 17:09:15.146635262 +0000 UTC m=+25.818963185"
Jul 28 17:09:15.866988 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:15.866952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:15.867129 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:15.867082    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:16.866622 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:16.866453    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:16.867134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:16.866456    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:16.867134 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:16.866689    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:16.867134 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:16.866759    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:17.128004 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:17.127944    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:09:17.128369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:17.128325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"b0442dbb85a4c58fe548b42149f40d3f0ba975d7c835c20ff98dc57055e1b522"}
Jul 28 17:09:17.128583 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:17.128568    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:09:17.128741 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:17.128724    2560 scope.go:117] "RemoveContainer" containerID="ebe72ad3ca799695ca6823e52597abfc4087af53b456160f1c924a7c36d9b98f"
Jul 28 17:09:17.130157 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:17.130135    2560 generic.go:356] "Generic (PLEG): container finished" podID="523beaef-da25-4143-890d-e6eab2507a01" containerID="a92eceeba0a531889a5b483a917b897281935ef9abfdaf75b33af21818e46890" exitCode=0
Jul 28 17:09:17.130256 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:17.130158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4pf" event={"ID":"523beaef-da25-4143-890d-e6eab2507a01","Type":"ContainerDied","Data":"a92eceeba0a531889a5b483a917b897281935ef9abfdaf75b33af21818e46890"}
Jul 28 17:09:17.144856 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:17.144839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:09:17.866978 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:17.866944    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:17.867464 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:17.867089    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:18.136073 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:18.136006    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:09:18.136410 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:18.136390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" event={"ID":"cb247d8e-6b1f-43c4-9e25-75636655445b","Type":"ContainerStarted","Data":"ac6fc0ffb4dfca1f0d5646be560a1aba0beb0de9e6759b231b689a4722c70e3c"}
Jul 28 17:09:18.136703 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:18.136679    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:09:18.136830 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:18.136759    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:09:18.151555 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:18.151538    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:09:18.170007 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:18.169958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5" podStartSLOduration=9.369848135 podStartE2EDuration="29.169941224s" totalImagesPullingTime="19.800093089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:18.16943268 +0000 UTC m=+28.841760602" watchObservedRunningTime="2026-07-28 17:09:18.169941224 +0000 UTC m=+28.842269147"
Jul 28 17:09:18.866932 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:18.866903    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:18.867075 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:18.866907    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:18.867313 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:18.866990    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:18.867313 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:18.867077    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:19.140518 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:19.140447    2560 generic.go:356] "Generic (PLEG): container finished" podID="523beaef-da25-4143-890d-e6eab2507a01" containerID="ad2989293dfdec6d6392769d24f3c605c44b2d589870f0ee1cac513b8881371a" exitCode=0
Jul 28 17:09:19.140644 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:19.140530    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4pf" event={"ID":"523beaef-da25-4143-890d-e6eab2507a01","Type":"ContainerDied","Data":"ad2989293dfdec6d6392769d24f3c605c44b2d589870f0ee1cac513b8881371a"}
Jul 28 17:09:19.867356 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:19.867316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:19.867712 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:19.867421    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:20.866752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:20.866724    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:20.866903 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:20.866809    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:20.866903 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:20.866862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:20.866990 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:20.866940    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:21.146257 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:21.146182    2560 generic.go:356] "Generic (PLEG): container finished" podID="523beaef-da25-4143-890d-e6eab2507a01" containerID="f58e9b2a22b5676acc1fb8776a7958ae30a90fdbe8b321247206811b4ee806bf" exitCode=0
Jul 28 17:09:21.146257 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:21.146220    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4pf" event={"ID":"523beaef-da25-4143-890d-e6eab2507a01","Type":"ContainerDied","Data":"f58e9b2a22b5676acc1fb8776a7958ae30a90fdbe8b321247206811b4ee806bf"}
Jul 28 17:09:21.866993 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:21.866952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:21.867172 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:21.867067    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:22.465337 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:22.465301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:22.465774 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.465475    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:22.465774 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.465552    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs podName:3aaf58b5-67e4-4a7f-b5fe-472d3968101e nodeName:}" failed. No retries permitted until 2026-07-28 17:09:54.465530453 +0000 UTC m=+65.137858357 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs") pod "network-metrics-daemon-n655v" (UID: "3aaf58b5-67e4-4a7f-b5fe-472d3968101e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:22.565814 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:22.565786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:22.565949 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.565884    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:22.565949 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.565929    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret podName:b207869a-8b53-42dd-a318-2645c2d55a3d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:54.565917634 +0000 UTC m=+65.238245533 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret") pod "global-pull-secret-syncer-ng8vd" (UID: "b207869a-8b53-42dd-a318-2645c2d55a3d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:22.666407 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:22.666376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:22.666587 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.666558    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:22.666587 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.666578    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:22.666676 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.666591    2560 projected.go:196] Error preparing data for projected volume kube-api-access-qg5qr for pod openshift-network-diagnostics/network-check-target-jxsd6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:22.666676 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.666643    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr podName:1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:54.666625905 +0000 UTC m=+65.338953807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qg5qr" (UniqueName: "kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr") pod "network-check-target-jxsd6" (UID: "1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:22.867175 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:22.867130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:22.867371 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.867259    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:22.867371 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:22.867317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:22.867478 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:22.867439    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:23.866957 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:23.866913    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:23.867410 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:23.867068    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:24.866435 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:24.866404    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:24.866435 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:24.866436    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:24.866690 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:24.866550    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:24.866690 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:24.866618    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:25.872061 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:25.872036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:25.872427 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:25.872134    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:26.160821 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:26.160738    2560 generic.go:356] "Generic (PLEG): container finished" podID="523beaef-da25-4143-890d-e6eab2507a01" containerID="55878127206e7768d5d2470786f24e10460ecc1e0cd8e6a50fdf853132648c1e" exitCode=0
Jul 28 17:09:26.160821 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:26.160776    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4pf" event={"ID":"523beaef-da25-4143-890d-e6eab2507a01","Type":"ContainerDied","Data":"55878127206e7768d5d2470786f24e10460ecc1e0cd8e6a50fdf853132648c1e"}
Jul 28 17:09:26.866592 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:26.866547    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:26.866592 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:26.866595    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:26.866785 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:26.866660    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:26.866822 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:26.866775    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:27.165152 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:27.165071    2560 generic.go:356] "Generic (PLEG): container finished" podID="523beaef-da25-4143-890d-e6eab2507a01" containerID="33fddd46cf001128ffa5a0bb29c2ee9fb07d6713a413cb2082bd5adaccafc267" exitCode=0
Jul 28 17:09:27.165152 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:27.165114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4pf" event={"ID":"523beaef-da25-4143-890d-e6eab2507a01","Type":"ContainerDied","Data":"33fddd46cf001128ffa5a0bb29c2ee9fb07d6713a413cb2082bd5adaccafc267"}
Jul 28 17:09:27.869563 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:27.869534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:27.869737 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:27.869654    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:28.170494 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:28.170417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4pf" event={"ID":"523beaef-da25-4143-890d-e6eab2507a01","Type":"ContainerStarted","Data":"dac0355290cde29d7902b47e9533fff2aec856e5540b2b42c12927afbab1a96f"}
Jul 28 17:09:28.203296 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:28.203255    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-js4pf" podStartSLOduration=8.158099681 podStartE2EDuration="38.203241349s" totalImagesPullingTime="30.045141668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:28.202731354 +0000 UTC m=+38.875059274" watchObservedRunningTime="2026-07-28 17:09:28.203241349 +0000 UTC m=+38.875569269"
Jul 28 17:09:28.866522 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:28.866495    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:28.866683 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:28.866497    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:28.866683 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:28.866661    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:28.866683 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:28.866581    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:29.866982 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:29.866947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:29.867411 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:29.867052    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:30.867137 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:30.867109    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:30.867617 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:30.867109    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:30.867617 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:30.867306    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:30.867617 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:30.867199    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:31.866552 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:31.866509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:31.866710 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:31.866643    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:32.785803 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:32.785775    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n655v"]
Jul 28 17:09:32.786283 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:32.785894    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:32.786283 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:32.786018    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:32.787301 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:32.787267    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ng8vd"]
Jul 28 17:09:32.787458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:32.787400    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:32.787525 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:32.787496    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:32.788369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:32.788326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxsd6"]
Jul 28 17:09:32.788461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:32.788430    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:32.788541 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:32.788519    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:33.867285 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:33.867119    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:33.867732 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:33.867403    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:34.866282 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:34.866260    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:34.866516 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:34.866263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:34.866516 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:34.866366    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ng8vd" podUID="b207869a-8b53-42dd-a318-2645c2d55a3d"
Jul 28 17:09:34.866516 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:34.866428    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxsd6" podUID="1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d"
Jul 28 17:09:35.866858 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:35.866822    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:35.867231 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:35.866959    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n655v" podUID="3aaf58b5-67e4-4a7f-b5fe-472d3968101e"
Jul 28 17:09:36.687086 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.687060    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-222.ec2.internal" event="NodeReady"
Jul 28 17:09:36.687240 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.687166    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:09:36.760401 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.760299    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m7vsp"]
Jul 28 17:09:36.792609 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.792582    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rpj8m"]
Jul 28 17:09:36.792749 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.792689    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.795620 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.795594    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:09:36.795780 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.795630    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:09:36.795780 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.795630    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:09:36.795780 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.795742    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:09:36.795950 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.795744    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k4zqq\""
Jul 28 17:09:36.807937 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.807914    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m7vsp"]
Jul 28 17:09:36.808043 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.807976    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rpj8m"]
Jul 28 17:09:36.808109 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.808067    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.810300 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.810280    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pz4fk\""
Jul 28 17:09:36.810430 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.810289    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:09:36.810430 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.810359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:09:36.855099 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.855078    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cpgsl"]
Jul 28 17:09:36.865665 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-crio-socket\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.865755 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jrq\" (UniqueName: \"kubernetes.io/projected/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-kube-api-access-t8jrq\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.865755 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-data-volume\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.865755 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865705    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-config-volume\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.865876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.865876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-metrics-tls\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.865959 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.865959 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865916    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-tmp-dir\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.866031 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.865938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9kd\" (UniqueName: \"kubernetes.io/projected/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-kube-api-access-4f9kd\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.876128 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.876109    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cpgsl"]
Jul 28 17:09:36.876574 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.876191    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:36.876574 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.876197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:36.876574 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.876213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpgsl"
Jul 28 17:09:36.879170 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.879153    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:09:36.879261 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.879178    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:09:36.879261 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.879209    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hr5cl\""
Jul 28 17:09:36.879382 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.879323    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4m279\""
Jul 28 17:09:36.879571 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.879553    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:09:36.879652 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.879595    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:09:36.879652 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.879556    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:09:36.879754 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.879597    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:09:36.966211 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.966302 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-tmp-dir\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.966302 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9kd\" (UniqueName: \"kubernetes.io/projected/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-kube-api-access-4f9kd\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.966302 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-crio-socket\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.966302 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jrq\" (UniqueName: \"kubernetes.io/projected/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-kube-api-access-t8jrq\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.966302 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-data-volume\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.966584 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-config-volume\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.966584 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-crio-socket\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.966689 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-tmp-dir\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.966689 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966641    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787aaed5-ae9d-4db2-8961-331a362b78ea-cert\") pod \"ingress-canary-cpgsl\" (UID: \"787aaed5-ae9d-4db2-8961-331a362b78ea\") " pod="openshift-ingress-canary/ingress-canary-cpgsl"
Jul 28 17:09:36.966789 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.966789 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966726    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-data-volume\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.966895 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.966895 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-metrics-tls\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.966895 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966869    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwflc\" (UniqueName: \"kubernetes.io/projected/787aaed5-ae9d-4db2-8961-331a362b78ea-kube-api-access-wwflc\") pod \"ingress-canary-cpgsl\" (UID: \"787aaed5-ae9d-4db2-8961-331a362b78ea\") " pod="openshift-ingress-canary/ingress-canary-cpgsl"
Jul 28 17:09:36.967018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.966907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-config-volume\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.970335 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.970315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-metrics-tls\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.970433 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.970403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:36.976885 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.976866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jrq\" (UniqueName: \"kubernetes.io/projected/0c999d6c-61a0-41cd-bdf9-5e11b3397f50-kube-api-access-t8jrq\") pod \"dns-default-rpj8m\" (UID: \"0c999d6c-61a0-41cd-bdf9-5e11b3397f50\") " pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:36.977319 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:36.977299    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9kd\" (UniqueName: \"kubernetes.io/projected/25984382-4f6c-4bc6-ac92-8bf23d4c25b3-kube-api-access-4f9kd\") pod \"insights-runtime-extractor-m7vsp\" (UID: \"25984382-4f6c-4bc6-ac92-8bf23d4c25b3\") " pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:37.067056 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.067036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787aaed5-ae9d-4db2-8961-331a362b78ea-cert\") pod \"ingress-canary-cpgsl\" (UID: \"787aaed5-ae9d-4db2-8961-331a362b78ea\") " pod="openshift-ingress-canary/ingress-canary-cpgsl"
Jul 28 17:09:37.067145 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.067085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwflc\" (UniqueName: \"kubernetes.io/projected/787aaed5-ae9d-4db2-8961-331a362b78ea-kube-api-access-wwflc\") pod \"ingress-canary-cpgsl\" (UID: \"787aaed5-ae9d-4db2-8961-331a362b78ea\") " pod="openshift-ingress-canary/ingress-canary-cpgsl"
Jul 28 17:09:37.068966 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.068950    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787aaed5-ae9d-4db2-8961-331a362b78ea-cert\") pod \"ingress-canary-cpgsl\" (UID: \"787aaed5-ae9d-4db2-8961-331a362b78ea\") " pod="openshift-ingress-canary/ingress-canary-cpgsl"
Jul 28 17:09:37.077957 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.077936    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwflc\" (UniqueName: \"kubernetes.io/projected/787aaed5-ae9d-4db2-8961-331a362b78ea-kube-api-access-wwflc\") pod \"ingress-canary-cpgsl\" (UID: \"787aaed5-ae9d-4db2-8961-331a362b78ea\") " pod="openshift-ingress-canary/ingress-canary-cpgsl"
Jul 28 17:09:37.102618 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.102596    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m7vsp"
Jul 28 17:09:37.115742 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.115725    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:37.190308 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.190275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpgsl"
Jul 28 17:09:37.332728 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.332656    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rpj8m"]
Jul 28 17:09:37.335837 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.335802    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m7vsp"]
Jul 28 17:09:37.338161 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:37.338116    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c999d6c_61a0_41cd_bdf9_5e11b3397f50.slice/crio-2f810aecd5194dcafef84f3d43b411e62b273e754df6aff2082e3074f9cd2dc3 WatchSource:0}: Error finding container 2f810aecd5194dcafef84f3d43b411e62b273e754df6aff2082e3074f9cd2dc3: Status 404 returned error can't find the container with id 2f810aecd5194dcafef84f3d43b411e62b273e754df6aff2082e3074f9cd2dc3
Jul 28 17:09:37.340146 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:37.340118    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25984382_4f6c_4bc6_ac92_8bf23d4c25b3.slice/crio-16826b7230954fb547de778fe38e77fe4831aaf9efc85a6ac1ae88b9dc12d9ae WatchSource:0}: Error finding container 16826b7230954fb547de778fe38e77fe4831aaf9efc85a6ac1ae88b9dc12d9ae: Status 404 returned error can't find the container with id 16826b7230954fb547de778fe38e77fe4831aaf9efc85a6ac1ae88b9dc12d9ae
Jul 28 17:09:37.348922 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.348901    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cpgsl"]
Jul 28 17:09:37.352072 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:37.352053    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787aaed5_ae9d_4db2_8961_331a362b78ea.slice/crio-7f4a9b83f0573f5f9410c3053b9da6b64bb6ca2a1a44078be4325f479b6e98d6 WatchSource:0}: Error finding container 7f4a9b83f0573f5f9410c3053b9da6b64bb6ca2a1a44078be4325f479b6e98d6: Status 404 returned error can't find the container with id 7f4a9b83f0573f5f9410c3053b9da6b64bb6ca2a1a44078be4325f479b6e98d6
Jul 28 17:09:37.866321 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.866288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:37.869486 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.869464    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hnfw\""
Jul 28 17:09:37.869486 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:37.869464    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:09:38.162764 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.162485    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"]
Jul 28 17:09:38.176393 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.176367    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"
Jul 28 17:09:38.176752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.176731    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"]
Jul 28 17:09:38.179950 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.179925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hrwg6\""
Jul 28 17:09:38.181732 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.181678    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 17:09:38.198556 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.198507    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rpj8m" event={"ID":"0c999d6c-61a0-41cd-bdf9-5e11b3397f50","Type":"ContainerStarted","Data":"2f810aecd5194dcafef84f3d43b411e62b273e754df6aff2082e3074f9cd2dc3"}
Jul 28 17:09:38.200377 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.200334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7vsp" event={"ID":"25984382-4f6c-4bc6-ac92-8bf23d4c25b3","Type":"ContainerStarted","Data":"16826b7230954fb547de778fe38e77fe4831aaf9efc85a6ac1ae88b9dc12d9ae"}
Jul 28 17:09:38.201971 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.201937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cpgsl" event={"ID":"787aaed5-ae9d-4db2-8961-331a362b78ea","Type":"ContainerStarted","Data":"7f4a9b83f0573f5f9410c3053b9da6b64bb6ca2a1a44078be4325f479b6e98d6"}
Jul 28 17:09:38.275814 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.275740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/521ad204-242e-49c2-9c78-8b7d19da3fd8-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9hsw6\" (UID: \"521ad204-242e-49c2-9c78-8b7d19da3fd8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"
Jul 28 17:09:38.376614 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.376581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/521ad204-242e-49c2-9c78-8b7d19da3fd8-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9hsw6\" (UID: \"521ad204-242e-49c2-9c78-8b7d19da3fd8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"
Jul 28 17:09:38.385689 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.385662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/521ad204-242e-49c2-9c78-8b7d19da3fd8-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9hsw6\" (UID: \"521ad204-242e-49c2-9c78-8b7d19da3fd8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"
Jul 28 17:09:38.492985 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.492750    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"
Jul 28 17:09:38.662955 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:38.662880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"]
Jul 28 17:09:38.666901 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:38.666681    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521ad204_242e_49c2_9c78_8b7d19da3fd8.slice/crio-6aa9cd07f662001bff3b6db816427c8ecb89c6afc9b4f78375dfeb607df2125a WatchSource:0}: Error finding container 6aa9cd07f662001bff3b6db816427c8ecb89c6afc9b4f78375dfeb607df2125a: Status 404 returned error can't find the container with id 6aa9cd07f662001bff3b6db816427c8ecb89c6afc9b4f78375dfeb607df2125a
Jul 28 17:09:39.207162 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:39.207089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6" event={"ID":"521ad204-242e-49c2-9c78-8b7d19da3fd8","Type":"ContainerStarted","Data":"6aa9cd07f662001bff3b6db816427c8ecb89c6afc9b4f78375dfeb607df2125a"}
Jul 28 17:09:45.227193 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.227006    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6" event={"ID":"521ad204-242e-49c2-9c78-8b7d19da3fd8","Type":"ContainerStarted","Data":"71d3daca5480f77ab82011fdc458191c00fd501a6dd8d8bd7da398e37f90a164"}
Jul 28 17:09:45.228082 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.227250    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"
Jul 28 17:09:45.228624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.228594    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7vsp" event={"ID":"25984382-4f6c-4bc6-ac92-8bf23d4c25b3","Type":"ContainerStarted","Data":"6ed53d23eae1ac115a581fa73721b0c9dadf2671eedc581c9c26b95181687e23"}
Jul 28 17:09:45.230309 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.230286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rpj8m" event={"ID":"0c999d6c-61a0-41cd-bdf9-5e11b3397f50","Type":"ContainerStarted","Data":"6a4b5ef6e7886ed8d2f7840cb4417895db2beb43a9112e85a6af5dfd37ce8fd4"}
Jul 28 17:09:45.230398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.230332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rpj8m" event={"ID":"0c999d6c-61a0-41cd-bdf9-5e11b3397f50","Type":"ContainerStarted","Data":"865456bc52db1b1e70798e06a127d7857cd8ebc9eca1debddce8b32a4fa35695"}
Jul 28 17:09:45.230398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.230373    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:45.231696 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.231673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cpgsl" event={"ID":"787aaed5-ae9d-4db2-8961-331a362b78ea","Type":"ContainerStarted","Data":"aa2ad90f0a2e157a525313681b661fca11aab012270582dff59e805136a83aaf"}
Jul 28 17:09:45.232269 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.232255    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6"
Jul 28 17:09:45.282118 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.282078    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cpgsl" podStartSLOduration=2.23809273 podStartE2EDuration="9.282065828s" totalImagesPullingTime="7.043973098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:45.281431542 +0000 UTC m=+55.953759462" watchObservedRunningTime="2026-07-28 17:09:45.282065828 +0000 UTC m=+55.954393727"
Jul 28 17:09:45.282230 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.282171    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hsw6" podStartSLOduration=1.555442394 podStartE2EDuration="7.282164569s" totalImagesPullingTime="5.726722175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:45.259727723 +0000 UTC m=+55.932055645" watchObservedRunningTime="2026-07-28 17:09:45.282164569 +0000 UTC m=+55.954492491"
Jul 28 17:09:45.306240 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:45.306205    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rpj8m" podStartSLOduration=3.021257723 podStartE2EDuration="9.306194814s" totalImagesPullingTime="6.284937091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:45.3049607 +0000 UTC m=+55.977288623" watchObservedRunningTime="2026-07-28 17:09:45.306194814 +0000 UTC m=+55.978522713"
Jul 28 17:09:46.240808 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.240768    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"]
Jul 28 17:09:46.277843 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.276633    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.282928 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.280957    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:09:46.283203 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.283174    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l2w6z\""
Jul 28 17:09:46.283301 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.283213    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 17:09:46.283301 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.283280    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 17:09:46.283434 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.283174    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:09:46.283600 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.283582    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:09:46.287513 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.287455    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"]
Jul 28 17:09:46.339032 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.339000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e73c443e-0048-4555-be22-9a6ecadf5982-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.339171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.339066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47lr\" (UniqueName: \"kubernetes.io/projected/e73c443e-0048-4555-be22-9a6ecadf5982-kube-api-access-j47lr\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.339171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.339148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e73c443e-0048-4555-be22-9a6ecadf5982-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.339293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.339175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e73c443e-0048-4555-be22-9a6ecadf5982-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.440283 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.440250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e73c443e-0048-4555-be22-9a6ecadf5982-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.440454 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.440293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47lr\" (UniqueName: \"kubernetes.io/projected/e73c443e-0048-4555-be22-9a6ecadf5982-kube-api-access-j47lr\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.440454 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.440376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e73c443e-0048-4555-be22-9a6ecadf5982-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.440454 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.440403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e73c443e-0048-4555-be22-9a6ecadf5982-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.441129 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.441103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e73c443e-0048-4555-be22-9a6ecadf5982-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.444234 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.444212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e73c443e-0048-4555-be22-9a6ecadf5982-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.444326 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.444214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e73c443e-0048-4555-be22-9a6ecadf5982-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.450838 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.450814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47lr\" (UniqueName: \"kubernetes.io/projected/e73c443e-0048-4555-be22-9a6ecadf5982-kube-api-access-j47lr\") pod \"prometheus-operator-84d69f794d-h2kgl\" (UID: \"e73c443e-0048-4555-be22-9a6ecadf5982\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:46.591074 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:46.591045    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"
Jul 28 17:09:48.564840 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:48.564811    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-h2kgl"]
Jul 28 17:09:48.568967 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:48.568943    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73c443e_0048_4555_be22_9a6ecadf5982.slice/crio-f8824b45b56ca762e758d178fea28fd84f026de18f81daf0cf34b092ae5554ab WatchSource:0}: Error finding container f8824b45b56ca762e758d178fea28fd84f026de18f81daf0cf34b092ae5554ab: Status 404 returned error can't find the container with id f8824b45b56ca762e758d178fea28fd84f026de18f81daf0cf34b092ae5554ab
Jul 28 17:09:49.248338 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:49.248301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7vsp" event={"ID":"25984382-4f6c-4bc6-ac92-8bf23d4c25b3","Type":"ContainerStarted","Data":"04986af4c2d90dd5bdf6897c4c135e19402b31c65a4858744dc10ef5d7340a9a"}
Jul 28 17:09:49.249707 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:49.249677    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl" event={"ID":"e73c443e-0048-4555-be22-9a6ecadf5982","Type":"ContainerStarted","Data":"f8824b45b56ca762e758d178fea28fd84f026de18f81daf0cf34b092ae5554ab"}
Jul 28 17:09:50.152463 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:50.152436    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c8k5"
Jul 28 17:09:50.257066 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:50.256238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl" event={"ID":"e73c443e-0048-4555-be22-9a6ecadf5982","Type":"ContainerStarted","Data":"630e7fed7bfb4fc85c364bca3836ef6811e59ee3bfda2fc0b7306e76a0009ee1"}
Jul 28 17:09:50.257066 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:50.256358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl" event={"ID":"e73c443e-0048-4555-be22-9a6ecadf5982","Type":"ContainerStarted","Data":"9a934b9578fa4fadbb4e39bfd82d38f80e1014203846081c9f1283cffa57237b"}
Jul 28 17:09:50.279667 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:50.279626    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-h2kgl" podStartSLOduration=2.858975382 podStartE2EDuration="4.27961059s" totalImagesPullingTime="1.420635208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:50.279299859 +0000 UTC m=+60.951627781" watchObservedRunningTime="2026-07-28 17:09:50.27961059 +0000 UTC m=+60.951938528"
Jul 28 17:09:52.613827 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.613793    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"]
Jul 28 17:09:52.616701 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.616684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.619787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.619767    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 17:09:52.619900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.619781    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:09:52.619900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.619784    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7dkpn\""
Jul 28 17:09:52.630112 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.630093    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"]
Jul 28 17:09:52.633194 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.633168    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vb65p"]
Jul 28 17:09:52.634914 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.634900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.638021 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.638003    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5mp2x\""
Jul 28 17:09:52.638268 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.638252    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:09:52.638596 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.638577    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:09:52.638682 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.638582    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:09:52.638682 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.638606    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"]
Jul 28 17:09:52.640576 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.640558    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.642468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.642453    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 17:09:52.642974 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.642958    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 17:09:52.643227 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.643212    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:09:52.643462 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.643447    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cmgxh\""
Jul 28 17:09:52.659591 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.659575    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"]
Jul 28 17:09:52.686185 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f516ae8-d603-4551-ad56-571deee69bcc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.686276 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-sys\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686276 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-root\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686276 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-textfile\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686399 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686293    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.686399 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686327    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btn5p\" (UniqueName: \"kubernetes.io/projected/8f516ae8-d603-4551-ad56-571deee69bcc-kube-api-access-btn5p\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.686399 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-wtmp\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686399 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.686524 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/068170ef-4e9a-4bf3-bac3-466a630b4bb9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.686524 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.686585 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686534    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-tls\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686585 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686553    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.686646 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686581    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686646 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5p6\" (UniqueName: \"kubernetes.io/projected/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-kube-api-access-cl5p6\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686646 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.686736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-metrics-client-ca\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/068170ef-4e9a-4bf3-bac3-466a630b4bb9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.686797 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.686797 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.686766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxms\" (UniqueName: \"kubernetes.io/projected/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-api-access-2qxms\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.787105 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.787105 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btn5p\" (UniqueName: \"kubernetes.io/projected/8f516ae8-d603-4551-ad56-571deee69bcc-kube-api-access-btn5p\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.787287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-wtmp\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.787287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.787287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/068170ef-4e9a-4bf3-bac3-466a630b4bb9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.787287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.787287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787253    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-tls\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.787287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.787287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-wtmp\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:52.787252    2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:52.787383    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:52.787383    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-tls podName:068170ef-4e9a-4bf3-bac3-466a630b4bb9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:53.287336191 +0000 UTC m=+63.959664105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-k4qrf" (UID: "068170ef-4e9a-4bf3-bac3-466a630b4bb9") : secret "kube-state-metrics-tls" not found
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:52.787433    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-tls podName:8f516ae8-d603-4551-ad56-571deee69bcc nodeName:}" failed. No retries permitted until 2026-07-28 17:09:53.287422633 +0000 UTC m=+63.959750536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-2xtjr" (UID: "8f516ae8-d603-4551-ad56-571deee69bcc") : secret "openshift-state-metrics-tls" not found
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5p6\" (UniqueName: \"kubernetes.io/projected/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-kube-api-access-cl5p6\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-metrics-client-ca\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/068170ef-4e9a-4bf3-bac3-466a630b4bb9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxms\" (UniqueName: \"kubernetes.io/projected/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-api-access-2qxms\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.787681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/068170ef-4e9a-4bf3-bac3-466a630b4bb9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.788222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f516ae8-d603-4551-ad56-571deee69bcc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.788222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-sys\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.788222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-root\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.788222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-textfile\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.788222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.787991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.788222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.788103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-textfile\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.788529 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.788219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.788529 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.788271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/068170ef-4e9a-4bf3-bac3-466a630b4bb9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.788529 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.788455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-metrics-client-ca\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.788529 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.788519    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-sys\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.788700 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.788564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-root\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.788874 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.788859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f516ae8-d603-4551-ad56-571deee69bcc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.790113 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.790090    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-tls\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.790207 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.790151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.790362 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.790322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.790443 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.790333    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.807513 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.807494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5p6\" (UniqueName: \"kubernetes.io/projected/de5431c3-9478-45f6-8e5f-cb6d3476ebfb-kube-api-access-cl5p6\") pod \"node-exporter-vb65p\" (UID: \"de5431c3-9478-45f6-8e5f-cb6d3476ebfb\") " pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.807726 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.807708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btn5p\" (UniqueName: \"kubernetes.io/projected/8f516ae8-d603-4551-ad56-571deee69bcc-kube-api-access-btn5p\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:52.808870 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.808848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxms\" (UniqueName: \"kubernetes.io/projected/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-api-access-2qxms\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:52.943769 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:52.943711    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vb65p"
Jul 28 17:09:52.954617 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:52.954596    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5431c3_9478_45f6_8e5f_cb6d3476ebfb.slice/crio-de2c0d24f70806fc760d86c1e1662ded06d6126c75c2a244bd1bface9007337e WatchSource:0}: Error finding container de2c0d24f70806fc760d86c1e1662ded06d6126c75c2a244bd1bface9007337e: Status 404 returned error can't find the container with id de2c0d24f70806fc760d86c1e1662ded06d6126c75c2a244bd1bface9007337e
Jul 28 17:09:53.266155 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.266077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vb65p" event={"ID":"de5431c3-9478-45f6-8e5f-cb6d3476ebfb","Type":"ContainerStarted","Data":"de2c0d24f70806fc760d86c1e1662ded06d6126c75c2a244bd1bface9007337e"}
Jul 28 17:09:53.268206 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.268173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m7vsp" event={"ID":"25984382-4f6c-4bc6-ac92-8bf23d4c25b3","Type":"ContainerStarted","Data":"104e4bec3aa268c4dd3040ab84e2e11669e2af8d1a83056e8d34d52339505ae8"}
Jul 28 17:09:53.289750 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.289707    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m7vsp" podStartSLOduration=2.570778893 podStartE2EDuration="17.289692388s" totalImagesPullingTime="14.718913495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:53.288912364 +0000 UTC m=+63.961240284" watchObservedRunningTime="2026-07-28 17:09:53.289692388 +0000 UTC m=+63.962020286"
Jul 28 17:09:53.291577 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.291398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:53.292448 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.291659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:53.294530 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.294296    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/068170ef-4e9a-4bf3-bac3-466a630b4bb9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-k4qrf\" (UID: \"068170ef-4e9a-4bf3-bac3-466a630b4bb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:53.296108 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.296087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f516ae8-d603-4551-ad56-571deee69bcc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2xtjr\" (UID: \"8f516ae8-d603-4551-ad56-571deee69bcc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:53.525663 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.525601    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"
Jul 28 17:09:53.549743 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.549721    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"
Jul 28 17:09:53.670220 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.670193    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr"]
Jul 28 17:09:53.672581 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:53.672553    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f516ae8_d603_4551_ad56_571deee69bcc.slice/crio-ed494e3064ff4bb14439acfbe1149853ee86f6a6109a71589c470eb17adc5d44 WatchSource:0}: Error finding container ed494e3064ff4bb14439acfbe1149853ee86f6a6109a71589c470eb17adc5d44: Status 404 returned error can't find the container with id ed494e3064ff4bb14439acfbe1149853ee86f6a6109a71589c470eb17adc5d44
Jul 28 17:09:53.690241 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.690215    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf"]
Jul 28 17:09:53.700279 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:53.700253    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068170ef_4e9a_4bf3_bac3_466a630b4bb9.slice/crio-c0062d67e8d71a676bcd6a0298827a155ed13700ff4ee6aab0be49f4409397a6 WatchSource:0}: Error finding container c0062d67e8d71a676bcd6a0298827a155ed13700ff4ee6aab0be49f4409397a6: Status 404 returned error can't find the container with id c0062d67e8d71a676bcd6a0298827a155ed13700ff4ee6aab0be49f4409397a6
Jul 28 17:09:53.784355 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.784313    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:09:53.786437 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.786418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.789857 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.789841    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:09:53.790077 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.790048    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-j9lp7\""
Jul 28 17:09:53.790170 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.790136    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:09:53.790231 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.790207    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:09:53.790446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.790428    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:09:53.790446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.790444    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:09:53.790604 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.790507    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:09:53.790817 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.790800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:09:53.791052 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.791033    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:09:53.791239 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.791228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:09:53.812709 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.812688    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:09:53.896044 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smz57\" (UniqueName: \"kubernetes.io/projected/cdfb810c-c582-485a-ad8c-cc740d556e33-kube-api-access-smz57\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdfb810c-c582-485a-ad8c-cc740d556e33-config-out\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdfb810c-c582-485a-ad8c-cc740d556e33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb810c-c582-485a-ad8c-cc740d556e33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896291 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896291 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896291 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896291 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896291 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdfb810c-c582-485a-ad8c-cc740d556e33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896501 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdfb810c-c582-485a-ad8c-cc740d556e33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896501 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896501 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.896501 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.896395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-web-config\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997300 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997433 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdfb810c-c582-485a-ad8c-cc740d556e33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997433 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdfb810c-c582-485a-ad8c-cc740d556e33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997433 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997433 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997433 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-web-config\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997675 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smz57\" (UniqueName: \"kubernetes.io/projected/cdfb810c-c582-485a-ad8c-cc740d556e33-kube-api-access-smz57\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997675 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdfb810c-c582-485a-ad8c-cc740d556e33-config-out\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997675 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdfb810c-c582-485a-ad8c-cc740d556e33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997675 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb810c-c582-485a-ad8c-cc740d556e33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997675 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:53.997569    2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 28 17:09:53.997675 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997675 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.997675 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:53.997644    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-main-tls podName:cdfb810c-c582-485a-ad8c-cc740d556e33 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:54.497622741 +0000 UTC m=+65.169950662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "cdfb810c-c582-485a-ad8c-cc740d556e33") : secret "alertmanager-main-tls" not found
Jul 28 17:09:53.998038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.997688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.998879 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.998317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdfb810c-c582-485a-ad8c-cc740d556e33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.998879 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.998827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdfb810c-c582-485a-ad8c-cc740d556e33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:53.999478 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:53.999429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb810c-c582-485a-ad8c-cc740d556e33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.000332 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.000305    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.000440 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.000369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.000550 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.000527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.000637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.000568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.001003 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.000989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-web-config\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.001111 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.001086    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdfb810c-c582-485a-ad8c-cc740d556e33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.001326 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.001308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdfb810c-c582-485a-ad8c-cc740d556e33-config-out\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.001452 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.001435    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.006797 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.006775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smz57\" (UniqueName: \"kubernetes.io/projected/cdfb810c-c582-485a-ad8c-cc740d556e33-kube-api-access-smz57\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.272329 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.272295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf" event={"ID":"068170ef-4e9a-4bf3-bac3-466a630b4bb9","Type":"ContainerStarted","Data":"c0062d67e8d71a676bcd6a0298827a155ed13700ff4ee6aab0be49f4409397a6"}
Jul 28 17:09:54.274094 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.274071    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr" event={"ID":"8f516ae8-d603-4551-ad56-571deee69bcc","Type":"ContainerStarted","Data":"98bbf71fe22b0b343ae17663394a32bb5c02463cf8fbd2db3ab4b79cd0473dff"}
Jul 28 17:09:54.274168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.274121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr" event={"ID":"8f516ae8-d603-4551-ad56-571deee69bcc","Type":"ContainerStarted","Data":"52ddf4f2d84973b9fbbef385d5cf3b170ff57ce210d865bb51dc6cf89de46527"}
Jul 28 17:09:54.274168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.274143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr" event={"ID":"8f516ae8-d603-4551-ad56-571deee69bcc","Type":"ContainerStarted","Data":"ed494e3064ff4bb14439acfbe1149853ee86f6a6109a71589c470eb17adc5d44"}
Jul 28 17:09:54.502660 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.502623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:54.502831 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.502676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.505613 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.505580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdfb810c-c582-485a-ad8c-cc740d556e33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdfb810c-c582-485a-ad8c-cc740d556e33\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.505930 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.505795    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:09:54.515407 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.515379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaf58b5-67e4-4a7f-b5fe-472d3968101e-metrics-certs\") pod \"network-metrics-daemon-n655v\" (UID: \"3aaf58b5-67e4-4a7f-b5fe-472d3968101e\") " pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:54.604120 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.604087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:54.607679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.607642    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:09:54.617860 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.617822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b207869a-8b53-42dd-a318-2645c2d55a3d-original-pull-secret\") pod \"global-pull-secret-syncer-ng8vd\" (UID: \"b207869a-8b53-42dd-a318-2645c2d55a3d\") " pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:54.680731 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.680703    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hnfw\""
Jul 28 17:09:54.688928 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.688904    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n655v"
Jul 28 17:09:54.695264 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.695224    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:09:54.704993 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.704971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:54.707393 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.707367    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:09:54.719212 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.719191    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:09:54.728816 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.728794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg5qr\" (UniqueName: \"kubernetes.io/projected/1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d-kube-api-access-qg5qr\") pod \"network-check-target-jxsd6\" (UID: \"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d\") " pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:54.750720 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.750335    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-59d48785df-kz9ll"]
Jul 28 17:09:54.753217 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.753198    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.759406 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.759385    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 17:09:54.759596 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.759386    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 17:09:54.759596 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.759502    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7fm8oprjuainp\""
Jul 28 17:09:54.759900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.759874    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zdmks\""
Jul 28 17:09:54.759900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.759874    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 17:09:54.760074 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.759952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 17:09:54.760443 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.760270    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 17:09:54.772998 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.772976    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59d48785df-kz9ll"]
Jul 28 17:09:54.810050 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.808919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29mzf\" (UniqueName: \"kubernetes.io/projected/7638afb8-032f-4073-a841-d8e3b8a3e739-kube-api-access-29mzf\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.810050 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.808998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.810050 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.809032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.810050 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.809073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-grpc-tls\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.810050 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.809114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-tls\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.810050 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.809152    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.810050 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.809188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.810050 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.809235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7638afb8-032f-4073-a841-d8e3b8a3e739-metrics-client-ca\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.890727 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.890618    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hr5cl\""
Jul 28 17:09:54.895155 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.895129    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:09:54.898792 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.898768    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ng8vd"
Jul 28 17:09:54.912398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.910508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.912398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.910568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.912398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.910612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-grpc-tls\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.912398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.910652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-tls\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.912398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.910685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.912398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.910717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.912398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.910757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7638afb8-032f-4073-a841-d8e3b8a3e739-metrics-client-ca\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.912398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.910806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29mzf\" (UniqueName: \"kubernetes.io/projected/7638afb8-032f-4073-a841-d8e3b8a3e739-kube-api-access-29mzf\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.914858 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.914720    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.917138 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.916813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7638afb8-032f-4073-a841-d8e3b8a3e739-metrics-client-ca\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.923758 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.923710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29mzf\" (UniqueName: \"kubernetes.io/projected/7638afb8-032f-4073-a841-d8e3b8a3e739-kube-api-access-29mzf\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.924008 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.923982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-grpc-tls\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.924563 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.924503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.924563 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.924547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.924820 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.924789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:54.927115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:54.927049    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7638afb8-032f-4073-a841-d8e3b8a3e739-secret-thanos-querier-tls\") pod \"thanos-querier-59d48785df-kz9ll\" (UID: \"7638afb8-032f-4073-a841-d8e3b8a3e739\") " pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:55.004635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.004568    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n655v"]
Jul 28 17:09:55.016438 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:55.016364    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaf58b5_67e4_4a7f_b5fe_472d3968101e.slice/crio-0e1bed15354a4eb9014eea2aaef76ba3956f45182b7f31a437fa51d34aa18636 WatchSource:0}: Error finding container 0e1bed15354a4eb9014eea2aaef76ba3956f45182b7f31a437fa51d34aa18636: Status 404 returned error can't find the container with id 0e1bed15354a4eb9014eea2aaef76ba3956f45182b7f31a437fa51d34aa18636
Jul 28 17:09:55.047480 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.047377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:09:55.065744 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.065511    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:09:55.092209 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.092139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ng8vd"]
Jul 28 17:09:55.115814 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.115714    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxsd6"]
Jul 28 17:09:55.237833 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.237807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rpj8m"
Jul 28 17:09:55.279894 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.279852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n655v" event={"ID":"3aaf58b5-67e4-4a7f-b5fe-472d3968101e","Type":"ContainerStarted","Data":"0e1bed15354a4eb9014eea2aaef76ba3956f45182b7f31a437fa51d34aa18636"}
Jul 28 17:09:55.283004 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.282943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr" event={"ID":"8f516ae8-d603-4551-ad56-571deee69bcc","Type":"ContainerStarted","Data":"9257d5fa4455145e07c1854d5d66f09db862dc66d32bdb852e1d55708df9548b"}
Jul 28 17:09:55.330499 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.330430    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2xtjr" podStartSLOduration=2.254116539 podStartE2EDuration="3.330413053s" totalImagesPullingTime="1.076296514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:55.328215676 +0000 UTC m=+66.000543597" watchObservedRunningTime="2026-07-28 17:09:55.330413053 +0000 UTC m=+66.002740972"
Jul 28 17:09:55.367935 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:55.367875    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfb810c_c582_485a_ad8c_cc740d556e33.slice/crio-3586db50465c57753c1c4714c3e98e816b481b9f11046c21aa3a9c6453e6a21e WatchSource:0}: Error finding container 3586db50465c57753c1c4714c3e98e816b481b9f11046c21aa3a9c6453e6a21e: Status 404 returned error can't find the container with id 3586db50465c57753c1c4714c3e98e816b481b9f11046c21aa3a9c6453e6a21e
Jul 28 17:09:55.368948 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:55.368815    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4790af_1eaa_4be6_8e4f_b70e6e4e3f6d.slice/crio-61b3f261c24306f6fddb850ea7f4f79a211ccc1f85d5fa45f9e2c1de85363a24 WatchSource:0}: Error finding container 61b3f261c24306f6fddb850ea7f4f79a211ccc1f85d5fa45f9e2c1de85363a24: Status 404 returned error can't find the container with id 61b3f261c24306f6fddb850ea7f4f79a211ccc1f85d5fa45f9e2c1de85363a24
Jul 28 17:09:55.559320 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:55.559281    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb207869a_8b53_42dd_a318_2645c2d55a3d.slice/crio-c3733e938d9a9b2814be8ce5c21091f04196d6dbc6cd8a543027c51a0a329fd0 WatchSource:0}: Error finding container c3733e938d9a9b2814be8ce5c21091f04196d6dbc6cd8a543027c51a0a329fd0: Status 404 returned error can't find the container with id c3733e938d9a9b2814be8ce5c21091f04196d6dbc6cd8a543027c51a0a329fd0
Jul 28 17:09:55.765333 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:55.765225    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59d48785df-kz9ll"]
Jul 28 17:09:56.292922 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.292585    2560 generic.go:356] "Generic (PLEG): container finished" podID="de5431c3-9478-45f6-8e5f-cb6d3476ebfb" containerID="f21a924cb4bb3e731d9e66bfeca799080b6acf223b03483297d875be8294adb2" exitCode=0
Jul 28 17:09:56.292922 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.292678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vb65p" event={"ID":"de5431c3-9478-45f6-8e5f-cb6d3476ebfb","Type":"ContainerDied","Data":"f21a924cb4bb3e731d9e66bfeca799080b6acf223b03483297d875be8294adb2"}
Jul 28 17:09:56.297452 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.296742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf" event={"ID":"068170ef-4e9a-4bf3-bac3-466a630b4bb9","Type":"ContainerStarted","Data":"b6a22a6413bbc57975d822220ec3a437aef926a6702d24793f6cb07ee069bc7a"}
Jul 28 17:09:56.297452 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.296796    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf" event={"ID":"068170ef-4e9a-4bf3-bac3-466a630b4bb9","Type":"ContainerStarted","Data":"c1e564119b7268edcc61f13b9099e429b074b7167bb63cdde57cf70f72b34f8a"}
Jul 28 17:09:56.297452 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.296824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf" event={"ID":"068170ef-4e9a-4bf3-bac3-466a630b4bb9","Type":"ContainerStarted","Data":"d629a69bfbde33f3d069a26782488af9ec0981f15cb983357904d7d88661aff3"}
Jul 28 17:09:56.299574 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.299520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll" event={"ID":"7638afb8-032f-4073-a841-d8e3b8a3e739","Type":"ContainerStarted","Data":"575918627e8698c38456543fd5021f2581b99bb53ee3fe570dd748a2a4a8b33b"}
Jul 28 17:09:56.301294 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.301230    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ng8vd" event={"ID":"b207869a-8b53-42dd-a318-2645c2d55a3d","Type":"ContainerStarted","Data":"c3733e938d9a9b2814be8ce5c21091f04196d6dbc6cd8a543027c51a0a329fd0"}
Jul 28 17:09:56.302750 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.302664    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxsd6" event={"ID":"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d","Type":"ContainerStarted","Data":"61b3f261c24306f6fddb850ea7f4f79a211ccc1f85d5fa45f9e2c1de85363a24"}
Jul 28 17:09:56.304650 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.304620    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdfb810c-c582-485a-ad8c-cc740d556e33","Type":"ContainerStarted","Data":"3586db50465c57753c1c4714c3e98e816b481b9f11046c21aa3a9c6453e6a21e"}
Jul 28 17:09:56.344487 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:56.344432    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-k4qrf" podStartSLOduration=2.447105831 podStartE2EDuration="4.344412712s" totalImagesPullingTime="1.897306881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:56.342596687 +0000 UTC m=+67.014924609" watchObservedRunningTime="2026-07-28 17:09:56.344412712 +0000 UTC m=+67.016740635"
Jul 28 17:09:57.057900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.056947    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84c9684999-thqzp"]
Jul 28 17:09:57.059631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.058869    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.065025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.064478    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 17:09:57.065025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.064490    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 17:09:57.065025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.064500    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:09:57.065025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.064756    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8phgc\""
Jul 28 17:09:57.065025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.064770    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-527501ob60taq\""
Jul 28 17:09:57.065025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.064937    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 17:09:57.073328 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.073308    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84c9684999-thqzp"]
Jul 28 17:09:57.132852 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.132821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3e481fb0-2784-4036-a119-42fec2b0625a-metrics-server-audit-profiles\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.133006 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.132894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-secret-metrics-server-tls\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.133006 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.132928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e481fb0-2784-4036-a119-42fec2b0625a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.133006 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.132959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-client-ca-bundle\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.133006 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.132984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdr9h\" (UniqueName: \"kubernetes.io/projected/3e481fb0-2784-4036-a119-42fec2b0625a-kube-api-access-kdr9h\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.133214 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.133010    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-secret-metrics-server-client-certs\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.133214 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.133075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3e481fb0-2784-4036-a119-42fec2b0625a-audit-log\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.234173 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.234140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e481fb0-2784-4036-a119-42fec2b0625a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.234331 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.234187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-client-ca-bundle\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.234331 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.234212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdr9h\" (UniqueName: \"kubernetes.io/projected/3e481fb0-2784-4036-a119-42fec2b0625a-kube-api-access-kdr9h\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.234331 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.234241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-secret-metrics-server-client-certs\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.234331 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.234304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3e481fb0-2784-4036-a119-42fec2b0625a-audit-log\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.234633 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.234370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3e481fb0-2784-4036-a119-42fec2b0625a-metrics-server-audit-profiles\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.234633 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.234432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-secret-metrics-server-tls\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.235127 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.235103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e481fb0-2784-4036-a119-42fec2b0625a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.235602 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.235554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3e481fb0-2784-4036-a119-42fec2b0625a-metrics-server-audit-profiles\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.235856 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.235829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3e481fb0-2784-4036-a119-42fec2b0625a-audit-log\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.238376 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.238297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-secret-metrics-server-client-certs\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.240008 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.239914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-client-ca-bundle\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.240416 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.240393    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3e481fb0-2784-4036-a119-42fec2b0625a-secret-metrics-server-tls\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.261242 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.261217    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdr9h\" (UniqueName: \"kubernetes.io/projected/3e481fb0-2784-4036-a119-42fec2b0625a-kube-api-access-kdr9h\") pod \"metrics-server-84c9684999-thqzp\" (UID: \"3e481fb0-2784-4036-a119-42fec2b0625a\") " pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.313299 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.313183    2560 generic.go:356] "Generic (PLEG): container finished" podID="cdfb810c-c582-485a-ad8c-cc740d556e33" containerID="a5cd5b70d71dd4d6b6919fe5d6c05af86284ce7524072269323d0f36bc8e9d87" exitCode=0
Jul 28 17:09:57.313299 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.313291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdfb810c-c582-485a-ad8c-cc740d556e33","Type":"ContainerDied","Data":"a5cd5b70d71dd4d6b6919fe5d6c05af86284ce7524072269323d0f36bc8e9d87"}
Jul 28 17:09:57.317211 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.316576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vb65p" event={"ID":"de5431c3-9478-45f6-8e5f-cb6d3476ebfb","Type":"ContainerStarted","Data":"c0a97063bbdc17fe909467cb3d020b3abf899e74cd54695610940a07bd0c9aa4"}
Jul 28 17:09:57.317211 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.316628    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vb65p" event={"ID":"de5431c3-9478-45f6-8e5f-cb6d3476ebfb","Type":"ContainerStarted","Data":"23956bc0c3afdfe9c4c996f07fa954d7437968842d61bad9d9fd3d653d08efe0"}
Jul 28 17:09:57.321019 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.320996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n655v" event={"ID":"3aaf58b5-67e4-4a7f-b5fe-472d3968101e","Type":"ContainerStarted","Data":"34fe48f7bb387fdd1d9e635423294a583ac03c5aae99c805f45c9118308aa22f"}
Jul 28 17:09:57.321160 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.321145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n655v" event={"ID":"3aaf58b5-67e4-4a7f-b5fe-472d3968101e","Type":"ContainerStarted","Data":"e89831e229119fdc2b2fd16d8f08e4771fe2fb15ec3a5bd078c70ca9aac6f3cf"}
Jul 28 17:09:57.372891 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.372862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:09:57.381964 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.381918    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n655v" podStartSLOduration=66.622546563 podStartE2EDuration="1m8.381898779s" totalImagesPullingTime="1.759352216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:57.380474668 +0000 UTC m=+68.052802591" watchObservedRunningTime="2026-07-28 17:09:57.381898779 +0000 UTC m=+68.054226701"
Jul 28 17:09:57.407262 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.406958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vb65p" podStartSLOduration=2.638891286 podStartE2EDuration="5.406938816s" totalImagesPullingTime="2.644132458s" totalInitContainerRuntime="123.915072ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:57.404823253 +0000 UTC m=+68.077151201" watchObservedRunningTime="2026-07-28 17:09:57.406938816 +0000 UTC m=+68.079266738"
Jul 28 17:09:57.459796 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.458855    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"]
Jul 28 17:09:57.466716 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.465894    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:57.470115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.469701    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 17:09:57.470115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.469972    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ssh5j\""
Jul 28 17:09:57.480084 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.479760    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"]
Jul 28 17:09:57.538413 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.538380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d3fb57c-3f51-467b-b726-20341c8d406a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2trmg\" (UID: \"5d3fb57c-3f51-467b-b726-20341c8d406a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:57.538601 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.538576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pm6\" (UniqueName: \"kubernetes.io/projected/5d3fb57c-3f51-467b-b726-20341c8d406a-kube-api-access-g5pm6\") pod \"monitoring-plugin-68b7fc9574-2trmg\" (UID: \"5d3fb57c-3f51-467b-b726-20341c8d406a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:57.558391 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.558296    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84c9684999-thqzp"]
Jul 28 17:09:57.563451 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:09:57.563385    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e481fb0_2784_4036_a119_42fec2b0625a.slice/crio-c285c8eb9ddf70142f150ee908b8e726de75d6e2abcd8ed237d31726964ba1fd WatchSource:0}: Error finding container c285c8eb9ddf70142f150ee908b8e726de75d6e2abcd8ed237d31726964ba1fd: Status 404 returned error can't find the container with id c285c8eb9ddf70142f150ee908b8e726de75d6e2abcd8ed237d31726964ba1fd
Jul 28 17:09:57.641666 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.640928    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pm6\" (UniqueName: \"kubernetes.io/projected/5d3fb57c-3f51-467b-b726-20341c8d406a-kube-api-access-g5pm6\") pod \"monitoring-plugin-68b7fc9574-2trmg\" (UID: \"5d3fb57c-3f51-467b-b726-20341c8d406a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:57.641666 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.640993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d3fb57c-3f51-467b-b726-20341c8d406a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2trmg\" (UID: \"5d3fb57c-3f51-467b-b726-20341c8d406a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:57.641666 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:57.641159    2560 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 17:09:57.641666 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:09:57.641224    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3fb57c-3f51-467b-b726-20341c8d406a-monitoring-plugin-cert podName:5d3fb57c-3f51-467b-b726-20341c8d406a nodeName:}" failed. No retries permitted until 2026-07-28 17:09:58.141202877 +0000 UTC m=+68.813530780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5d3fb57c-3f51-467b-b726-20341c8d406a-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-2trmg" (UID: "5d3fb57c-3f51-467b-b726-20341c8d406a") : secret "monitoring-plugin-cert" not found
Jul 28 17:09:57.653760 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:57.653711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pm6\" (UniqueName: \"kubernetes.io/projected/5d3fb57c-3f51-467b-b726-20341c8d406a-kube-api-access-g5pm6\") pod \"monitoring-plugin-68b7fc9574-2trmg\" (UID: \"5d3fb57c-3f51-467b-b726-20341c8d406a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:58.146867 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:58.146310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d3fb57c-3f51-467b-b726-20341c8d406a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2trmg\" (UID: \"5d3fb57c-3f51-467b-b726-20341c8d406a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:58.149947 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:58.149866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5d3fb57c-3f51-467b-b726-20341c8d406a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2trmg\" (UID: \"5d3fb57c-3f51-467b-b726-20341c8d406a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:58.327823 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:58.327743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84c9684999-thqzp" event={"ID":"3e481fb0-2784-4036-a119-42fec2b0625a","Type":"ContainerStarted","Data":"c285c8eb9ddf70142f150ee908b8e726de75d6e2abcd8ed237d31726964ba1fd"}
Jul 28 17:09:58.384989 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:58.384762    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:09:59.122474 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.122442    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:09:59.141871 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.141843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.156668 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.156642    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:09:59.157453 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.157436    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:09:59.157651 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.157637    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:09:59.158779 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.158755    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:09:59.159335 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.159314    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f42snapggce3b\""
Jul 28 17:09:59.159433 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.159337    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:09:59.159603 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.159586    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xbtzv\""
Jul 28 17:09:59.170553 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.170527    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:09:59.173369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.170746    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:09:59.173369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.170913    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:09:59.173369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.171098    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:09:59.173369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.171311    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:09:59.173369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.173078    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:09:59.173369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.173252    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:09:59.185123 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.185085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:09:59.191532 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.191499    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:09:59.192902 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.192779    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:09:59.260186 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-config\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260411 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260411 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260411 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260411 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p458g\" (UniqueName: \"kubernetes.io/projected/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-kube-api-access-p458g\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.260876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.261054 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.261054 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.261054 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.261054 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.260978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362477    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.362624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-config\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.362814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363401 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.363040    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.363843 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.363722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.364163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.363944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p458g\" (UniqueName: \"kubernetes.io/projected/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-kube-api-access-p458g\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.364163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.364005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.365058 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.365032    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.365728 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.365704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.366204 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.366177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.368161 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.366478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.368161 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.366946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.368161 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.367159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.369442 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.369420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.371373 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.371216    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-config\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.371373 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.371217    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.371373 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.371312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.371568 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.371407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.371703 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.371680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.372438 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.372371    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.372543 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.372465    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.372675 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.372649    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.375918 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.375711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.376519 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.376484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.382426 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.382403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p458g\" (UniqueName: \"kubernetes.io/projected/fd043a1e-c78e-4c16-baa2-2a8b3fa734cc-kube-api-access-p458g\") pod \"prometheus-k8s-0\" (UID: \"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:09:59.453435 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:09:59.453403    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:10:02.304612 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.304587    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:10:02.310387 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:10:02.310216    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd043a1e_c78e_4c16_baa2_2a8b3fa734cc.slice/crio-558879b9e21b79e3b5c81b400c5e7f9082a1c54659344091656f654e094d2a53 WatchSource:0}: Error finding container 558879b9e21b79e3b5c81b400c5e7f9082a1c54659344091656f654e094d2a53: Status 404 returned error can't find the container with id 558879b9e21b79e3b5c81b400c5e7f9082a1c54659344091656f654e094d2a53
Jul 28 17:10:02.321502 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.320945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"]
Jul 28 17:10:02.363532 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.363504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxsd6" event={"ID":"1d4790af-1eaa-4be6-8e4f-b70e6e4e3f6d","Type":"ContainerStarted","Data":"f49e3d83f0ae6a4423e593c72295d628a8fdbdcd2b8a31c0a3747803b373a7df"}
Jul 28 17:10:02.363975 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.363953    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:10:02.366135 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.366108    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdfb810c-c582-485a-ad8c-cc740d556e33","Type":"ContainerStarted","Data":"1819aac755df06aab960b427515b6338180456c949076dd70683a8b80b2b9e69"}
Jul 28 17:10:02.369087 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.369065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll" event={"ID":"7638afb8-032f-4073-a841-d8e3b8a3e739","Type":"ContainerStarted","Data":"ac35a2892996dccba43525a1b0471abd50df4d1a8790891590d173cf7022d9ec"}
Jul 28 17:10:02.369284 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.369256    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll" event={"ID":"7638afb8-032f-4073-a841-d8e3b8a3e739","Type":"ContainerStarted","Data":"9b0c4d1ef1003c2eadf37b626d8175127994c730d0db90e6a9bb05883ba51dc9"}
Jul 28 17:10:02.371078 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.371022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ng8vd" event={"ID":"b207869a-8b53-42dd-a318-2645c2d55a3d","Type":"ContainerStarted","Data":"e8c8a1cfa887cf170b963bb7907534efbd420a13fd9bfa9bae06c55287cf522a"}
Jul 28 17:10:02.378631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.376458    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84c9684999-thqzp" event={"ID":"3e481fb0-2784-4036-a119-42fec2b0625a","Type":"ContainerStarted","Data":"338d98739a0fe7ada02d28f9683c1308f3daeb8eb1a1301b03d5624302d3ad88"}
Jul 28 17:10:02.380189 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.380110    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc","Type":"ContainerStarted","Data":"558879b9e21b79e3b5c81b400c5e7f9082a1c54659344091656f654e094d2a53"}
Jul 28 17:10:02.381933 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.381894    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg" event={"ID":"5d3fb57c-3f51-467b-b726-20341c8d406a","Type":"ContainerStarted","Data":"93ead16df9d06ead9f0c01fd3512efc8b95ebba96d67c2123893c40ca317cf2e"}
Jul 28 17:10:02.390466 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.390425    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jxsd6" podStartSLOduration=65.642252185 podStartE2EDuration="1m12.390408799s" totalImagesPullingTime="6.748156614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:02.387543755 +0000 UTC m=+73.059871690" watchObservedRunningTime="2026-07-28 17:10:02.390408799 +0000 UTC m=+73.062736711"
Jul 28 17:10:02.441494 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.441452    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84c9684999-thqzp" podStartSLOduration=0.813206612 podStartE2EDuration="5.441433084s" totalImagesPullingTime="4.628226472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:02.440811425 +0000 UTC m=+73.113139370" watchObservedRunningTime="2026-07-28 17:10:02.441433084 +0000 UTC m=+73.113761005"
Jul 28 17:10:02.442033 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:02.441997    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ng8vd" podStartSLOduration=65.901742757 podStartE2EDuration="1m12.441986033s" totalImagesPullingTime="6.540243276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:02.40949559 +0000 UTC m=+73.081823515" watchObservedRunningTime="2026-07-28 17:10:02.441986033 +0000 UTC m=+73.114313953"
Jul 28 17:10:03.388813 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:03.388782    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdfb810c-c582-485a-ad8c-cc740d556e33","Type":"ContainerStarted","Data":"9afcb221edcc175a636ad0663d4b18db9077ce468fcfafb97196560495315c3f"}
Jul 28 17:10:03.389220 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:03.388874    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdfb810c-c582-485a-ad8c-cc740d556e33","Type":"ContainerStarted","Data":"c14951ece64d5b20382471440509825d8826980ef70b334f02fce6e86ff3f0d4"}
Jul 28 17:10:03.389220 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:03.388905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdfb810c-c582-485a-ad8c-cc740d556e33","Type":"ContainerStarted","Data":"136a8e7ff819cd1379720fabcbc79071b1668511aaff0a267b5341cacbfe72b0"}
Jul 28 17:10:03.389220 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:03.388930    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdfb810c-c582-485a-ad8c-cc740d556e33","Type":"ContainerStarted","Data":"9dc1c774dd2f48e35ab3ef6d78be3f936e78ed1dde62ba3eee2979246b1edc82"}
Jul 28 17:10:03.391160 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:03.391131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll" event={"ID":"7638afb8-032f-4073-a841-d8e3b8a3e739","Type":"ContainerStarted","Data":"12ec9826e54dd26908a904fb3c84877396c154fba890c769a06910a0d4878b94"}
Jul 28 17:10:03.392680 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:03.392659    2560 generic.go:356] "Generic (PLEG): container finished" podID="fd043a1e-c78e-4c16-baa2-2a8b3fa734cc" containerID="3547dfb24b290b9fc80568c9b5927ef740c4b3e68556d82f56f4f03bc9d4130d" exitCode=0
Jul 28 17:10:03.392787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:03.392761    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc","Type":"ContainerDied","Data":"3547dfb24b290b9fc80568c9b5927ef740c4b3e68556d82f56f4f03bc9d4130d"}
Jul 28 17:10:04.399765 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.399675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg" event={"ID":"5d3fb57c-3f51-467b-b726-20341c8d406a","Type":"ContainerStarted","Data":"fdb23ee51fd6c54f061438c2089806ec6a6d1a5928a72f47f87400c119d0f888"}
Jul 28 17:10:04.400192 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.399791    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:10:04.404057 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.404028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdfb810c-c582-485a-ad8c-cc740d556e33","Type":"ContainerStarted","Data":"9e7ccb551ac2cc6398bcc0e3199bf2845a75eaa13cc000c99fc194f9edfe4457"}
Jul 28 17:10:04.405942 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.405894    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg"
Jul 28 17:10:04.408045 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.408020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll" event={"ID":"7638afb8-032f-4073-a841-d8e3b8a3e739","Type":"ContainerStarted","Data":"a837b9cc6d748645ee54dc223bbaadfa59bb06b49db428f8accd5ed80d979478"}
Jul 28 17:10:04.408193 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.408080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll" event={"ID":"7638afb8-032f-4073-a841-d8e3b8a3e739","Type":"ContainerStarted","Data":"1a0dd539c7766abd344cfef569d6ddff4d1b2ee8c23574374dce3d040adf66ca"}
Jul 28 17:10:04.408193 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.408121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll" event={"ID":"7638afb8-032f-4073-a841-d8e3b8a3e739","Type":"ContainerStarted","Data":"61d096e3a1d1a542604fae510b9a2b570fff3eaceb6438433399e59acf520ea3"}
Jul 28 17:10:04.435819 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.435776    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2trmg" podStartSLOduration=6.039012861 podStartE2EDuration="7.435762023s" totalImagesPullingTime="1.396749162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:04.434310069 +0000 UTC m=+75.106637985" watchObservedRunningTime="2026-07-28 17:10:04.435762023 +0000 UTC m=+75.108089943"
Jul 28 17:10:04.484225 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.484180    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.144426796 podStartE2EDuration="11.48416708s" totalImagesPullingTime="7.314477208s" totalInitContainerRuntime="25.263076ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:04.483899933 +0000 UTC m=+75.156227854" watchObservedRunningTime="2026-07-28 17:10:04.48416708 +0000 UTC m=+75.156495023"
Jul 28 17:10:04.550991 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:04.550946    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll" podStartSLOduration=2.886138618 podStartE2EDuration="10.550925761s" totalImagesPullingTime="7.664787143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:04.550117033 +0000 UTC m=+75.222444955" watchObservedRunningTime="2026-07-28 17:10:04.550925761 +0000 UTC m=+75.223253683"
Jul 28 17:10:05.412476 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:05.412450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:10:07.423095 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:07.423017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc","Type":"ContainerStarted","Data":"cc6963ea6e2b18adfc0a0461a01ca04a4cb8dedaee98463db0cc6d4d6c7a7dcc"}
Jul 28 17:10:07.423446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:07.423130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc","Type":"ContainerStarted","Data":"f78cb97d2fca5dc2930992c62126e751ae9f9fd652b14316e46fdb838d24b958"}
Jul 28 17:10:07.423446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:07.423163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc","Type":"ContainerStarted","Data":"1699669c53fa469b2c003e5bf388d93429cbd259b56b671bd5f8d0f9e34ac428"}
Jul 28 17:10:07.423446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:07.423189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc","Type":"ContainerStarted","Data":"81df193367a7a390de7c3b9e47772657d0f8da8749b41674e99e86c494562d36"}
Jul 28 17:10:07.423446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:07.423213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc","Type":"ContainerStarted","Data":"d603cc0f5edd36757fcb7169341a17fd0bca50655d83c7c80aa24f850be3e0d0"}
Jul 28 17:10:07.423446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:07.423244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd043a1e-c78e-4c16-baa2-2a8b3fa734cc","Type":"ContainerStarted","Data":"02f74760ddc56d741dcc5d35af3732a605de2f0c41fb589d63732662316e6237"}
Jul 28 17:10:09.453592 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:09.453556    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:10:11.421414 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:11.421289    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59d48785df-kz9ll"
Jul 28 17:10:11.453145 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:11.453100    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.941791688 podStartE2EDuration="12.453048667s" totalImagesPullingTime="3.484339672s" totalInitContainerRuntime="26.917307ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:07.47958366 +0000 UTC m=+78.151911580" watchObservedRunningTime="2026-07-28 17:10:11.453048667 +0000 UTC m=+82.125376586"
Jul 28 17:10:17.373703 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:17.373663    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:10:17.374168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:17.373725    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:10:34.410782 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:34.410752    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jxsd6"
Jul 28 17:10:37.378486 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:37.378455    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:10:37.382159 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:37.382138    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84c9684999-thqzp"
Jul 28 17:10:59.453937 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:59.453904    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:10:59.475897 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:59.475867    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:10:59.669405 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:10:59.669381    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:58.712528 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.712487    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x8jq4"]
Jul 28 17:12:58.714647 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.714625    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:12:58.717913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.717894    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rzvxd\""
Jul 28 17:12:58.718673 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.718646    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 17:12:58.718807 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.718685    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 17:12:58.726444 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.726420    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x8jq4"]
Jul 28 17:12:58.803618 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.803580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmsv\" (UniqueName: \"kubernetes.io/projected/5c48929a-7c52-4afa-95af-205e3cf89da6-kube-api-access-hhmsv\") pod \"cert-manager-webhook-6486ffd997-x8jq4\" (UID: \"5c48929a-7c52-4afa-95af-205e3cf89da6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:12:58.803805 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.803654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c48929a-7c52-4afa-95af-205e3cf89da6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x8jq4\" (UID: \"5c48929a-7c52-4afa-95af-205e3cf89da6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:12:58.904833 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.904790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmsv\" (UniqueName: \"kubernetes.io/projected/5c48929a-7c52-4afa-95af-205e3cf89da6-kube-api-access-hhmsv\") pod \"cert-manager-webhook-6486ffd997-x8jq4\" (UID: \"5c48929a-7c52-4afa-95af-205e3cf89da6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:12:58.905008 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.904881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c48929a-7c52-4afa-95af-205e3cf89da6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x8jq4\" (UID: \"5c48929a-7c52-4afa-95af-205e3cf89da6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:12:58.918189 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.918152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c48929a-7c52-4afa-95af-205e3cf89da6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x8jq4\" (UID: \"5c48929a-7c52-4afa-95af-205e3cf89da6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:12:58.921701 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:58.921678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmsv\" (UniqueName: \"kubernetes.io/projected/5c48929a-7c52-4afa-95af-205e3cf89da6-kube-api-access-hhmsv\") pod \"cert-manager-webhook-6486ffd997-x8jq4\" (UID: \"5c48929a-7c52-4afa-95af-205e3cf89da6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:12:59.038125 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:59.038023    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:12:59.211654 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:12:59.211627    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x8jq4"]
Jul 28 17:12:59.213549 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:12:59.213504    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c48929a_7c52_4afa_95af_205e3cf89da6.slice/crio-2b2859c76adbf8b8fad2c11498de0f4c28e55701c3412ff1fdc9e092138857da WatchSource:0}: Error finding container 2b2859c76adbf8b8fad2c11498de0f4c28e55701c3412ff1fdc9e092138857da: Status 404 returned error can't find the container with id 2b2859c76adbf8b8fad2c11498de0f4c28e55701c3412ff1fdc9e092138857da
Jul 28 17:13:00.135029 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:00.134982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4" event={"ID":"5c48929a-7c52-4afa-95af-205e3cf89da6","Type":"ContainerStarted","Data":"2b2859c76adbf8b8fad2c11498de0f4c28e55701c3412ff1fdc9e092138857da"}
Jul 28 17:13:03.151000 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:03.150961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4" event={"ID":"5c48929a-7c52-4afa-95af-205e3cf89da6","Type":"ContainerStarted","Data":"4408c60f4f1fe51170decd2d6a02e77abfb1d8954d4fb7a0a8ea20876480532a"}
Jul 28 17:13:03.151417 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:03.151091    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:13:03.170744 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:03.170679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4" podStartSLOduration=1.729295317 podStartE2EDuration="5.170663997s" totalImagesPullingTime="3.44136868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:03.168838128 +0000 UTC m=+253.841166048" watchObservedRunningTime="2026-07-28 17:13:03.170663997 +0000 UTC m=+253.842991911"
Jul 28 17:13:09.157509 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:09.157471    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-x8jq4"
Jul 28 17:13:11.414480 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.414400    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nz6gk"]
Jul 28 17:13:11.417563 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.417544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nz6gk"
Jul 28 17:13:11.419970 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.419945    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9ltgn\""
Jul 28 17:13:11.428435 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.428413    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nz6gk"]
Jul 28 17:13:11.517705 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.517680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20f6d517-ed64-4863-b891-a9a17a520158-bound-sa-token\") pod \"cert-manager-55b4d954d5-nz6gk\" (UID: \"20f6d517-ed64-4863-b891-a9a17a520158\") " pod="cert-manager/cert-manager-55b4d954d5-nz6gk"
Jul 28 17:13:11.517834 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.517711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8glg\" (UniqueName: \"kubernetes.io/projected/20f6d517-ed64-4863-b891-a9a17a520158-kube-api-access-q8glg\") pod \"cert-manager-55b4d954d5-nz6gk\" (UID: \"20f6d517-ed64-4863-b891-a9a17a520158\") " pod="cert-manager/cert-manager-55b4d954d5-nz6gk"
Jul 28 17:13:11.619002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.618970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20f6d517-ed64-4863-b891-a9a17a520158-bound-sa-token\") pod \"cert-manager-55b4d954d5-nz6gk\" (UID: \"20f6d517-ed64-4863-b891-a9a17a520158\") " pod="cert-manager/cert-manager-55b4d954d5-nz6gk"
Jul 28 17:13:11.619002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.619008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8glg\" (UniqueName: \"kubernetes.io/projected/20f6d517-ed64-4863-b891-a9a17a520158-kube-api-access-q8glg\") pod \"cert-manager-55b4d954d5-nz6gk\" (UID: \"20f6d517-ed64-4863-b891-a9a17a520158\") " pod="cert-manager/cert-manager-55b4d954d5-nz6gk"
Jul 28 17:13:11.629203 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.629178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20f6d517-ed64-4863-b891-a9a17a520158-bound-sa-token\") pod \"cert-manager-55b4d954d5-nz6gk\" (UID: \"20f6d517-ed64-4863-b891-a9a17a520158\") " pod="cert-manager/cert-manager-55b4d954d5-nz6gk"
Jul 28 17:13:11.629328 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.629235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8glg\" (UniqueName: \"kubernetes.io/projected/20f6d517-ed64-4863-b891-a9a17a520158-kube-api-access-q8glg\") pod \"cert-manager-55b4d954d5-nz6gk\" (UID: \"20f6d517-ed64-4863-b891-a9a17a520158\") " pod="cert-manager/cert-manager-55b4d954d5-nz6gk"
Jul 28 17:13:11.726486 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.726407    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nz6gk"
Jul 28 17:13:11.862184 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:11.862159    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nz6gk"]
Jul 28 17:13:11.864520 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:13:11.864489    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20f6d517_ed64_4863_b891_a9a17a520158.slice/crio-ab9b144afaa75620aacd1903a3f008244ca05d834b679311d4af673f72f900c8 WatchSource:0}: Error finding container ab9b144afaa75620aacd1903a3f008244ca05d834b679311d4af673f72f900c8: Status 404 returned error can't find the container with id ab9b144afaa75620aacd1903a3f008244ca05d834b679311d4af673f72f900c8
Jul 28 17:13:12.187865 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:12.187832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nz6gk" event={"ID":"20f6d517-ed64-4863-b891-a9a17a520158","Type":"ContainerStarted","Data":"1685bbd9f6eb9f370fd24a91646c02126336add1055b41b2c62ab454591ade6b"}
Jul 28 17:13:12.187865 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:12.187873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nz6gk" event={"ID":"20f6d517-ed64-4863-b891-a9a17a520158","Type":"ContainerStarted","Data":"ab9b144afaa75620aacd1903a3f008244ca05d834b679311d4af673f72f900c8"}
Jul 28 17:13:12.210097 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:12.210032    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nz6gk" podStartSLOduration=1.210008451 podStartE2EDuration="1.210008451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:12.209553687 +0000 UTC m=+262.881881643" watchObservedRunningTime="2026-07-28 17:13:12.210008451 +0000 UTC m=+262.882336364"
Jul 28 17:13:32.225164 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.225131    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"]
Jul 28 17:13:32.229672 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.229644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.232460 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.232437    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 17:13:32.232679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.232665    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 17:13:32.233470 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.233436    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 17:13:32.238001 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.237981    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:13:32.240052 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.240034    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-njktv\""
Jul 28 17:13:32.240189 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.240172    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 17:13:32.242594 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.242569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"]
Jul 28 17:13:32.293964 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.293935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91095dca-a019-4981-9e98-0c45a1a9b5b7-metrics-cert\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.294161 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.293995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91095dca-a019-4981-9e98-0c45a1a9b5b7-cert\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.294161 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.294020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8vz\" (UniqueName: \"kubernetes.io/projected/91095dca-a019-4981-9e98-0c45a1a9b5b7-kube-api-access-kn8vz\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.294161 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.294077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91095dca-a019-4981-9e98-0c45a1a9b5b7-manager-config\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.395095 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.395053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91095dca-a019-4981-9e98-0c45a1a9b5b7-manager-config\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.395270 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.395148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91095dca-a019-4981-9e98-0c45a1a9b5b7-metrics-cert\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.395270 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.395196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91095dca-a019-4981-9e98-0c45a1a9b5b7-cert\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.395270 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.395221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8vz\" (UniqueName: \"kubernetes.io/projected/91095dca-a019-4981-9e98-0c45a1a9b5b7-kube-api-access-kn8vz\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.395869 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.395832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91095dca-a019-4981-9e98-0c45a1a9b5b7-manager-config\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.397657 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.397626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91095dca-a019-4981-9e98-0c45a1a9b5b7-cert\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.397733 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.397658    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/91095dca-a019-4981-9e98-0c45a1a9b5b7-metrics-cert\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.405080 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.405052    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8vz\" (UniqueName: \"kubernetes.io/projected/91095dca-a019-4981-9e98-0c45a1a9b5b7-kube-api-access-kn8vz\") pod \"lws-controller-manager-5cdc965c4d-rdf6p\" (UID: \"91095dca-a019-4981-9e98-0c45a1a9b5b7\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.540224 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.540136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:32.674225 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:32.674190    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"]
Jul 28 17:13:32.677968 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:13:32.677936    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91095dca_a019_4981_9e98_0c45a1a9b5b7.slice/crio-4072ae7b1d3c6c51048d443b946b86539ae0296ccb9d036745e79b142e19c26e WatchSource:0}: Error finding container 4072ae7b1d3c6c51048d443b946b86539ae0296ccb9d036745e79b142e19c26e: Status 404 returned error can't find the container with id 4072ae7b1d3c6c51048d443b946b86539ae0296ccb9d036745e79b142e19c26e
Jul 28 17:13:33.273403 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:33.273364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p" event={"ID":"91095dca-a019-4981-9e98-0c45a1a9b5b7","Type":"ContainerStarted","Data":"4072ae7b1d3c6c51048d443b946b86539ae0296ccb9d036745e79b142e19c26e"}
Jul 28 17:13:36.289665 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:36.289626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p" event={"ID":"91095dca-a019-4981-9e98-0c45a1a9b5b7","Type":"ContainerStarted","Data":"672243a96690dbe35b9bcb9ce51bd53db6d512e33374907f31a1598b03bdbc75"}
Jul 28 17:13:36.290016 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:36.289703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:36.311367 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:36.311292    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p" podStartSLOduration=0.840910613 podStartE2EDuration="4.311269764s" totalImagesPullingTime="3.470359151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:36.309686632 +0000 UTC m=+286.982014554" watchObservedRunningTime="2026-07-28 17:13:36.311269764 +0000 UTC m=+286.983597685"
Jul 28 17:13:44.200437 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.200401    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"]
Jul 28 17:13:44.203711 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.203686    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.208927 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.208894    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 17:13:44.209060 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.208932    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 17:13:44.209365 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.209322    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 17:13:44.209465 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.209363    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m9txd\""
Jul 28 17:13:44.209465 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.209380    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 17:13:44.242557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.242531    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"]
Jul 28 17:13:44.284570 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.284533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.284741 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.284587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8jg6\" (UniqueName: \"kubernetes.io/projected/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-kube-api-access-k8jg6\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.284741 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.284706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-manifests\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.284741 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.284736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.284896 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.284771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-tmp\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.386124 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.386085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8jg6\" (UniqueName: \"kubernetes.io/projected/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-kube-api-access-k8jg6\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.386306 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.386164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-manifests\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.386306 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.386185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.386306 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.386263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-tmp\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.386485 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.386310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.386694 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.386671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-manifests\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.388631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.388605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-tmp\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.388799 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.388781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.388842 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.388789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.394747 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.394721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8jg6\" (UniqueName: \"kubernetes.io/projected/22627ec9-dc12-4a35-9982-4e0fb1cdc9b5-kube-api-access-k8jg6\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh\" (UID: \"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.514052 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.513966    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:44.664150 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:44.664127    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"]
Jul 28 17:13:44.665821 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:13:44.665790    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22627ec9_dc12_4a35_9982_4e0fb1cdc9b5.slice/crio-3722e4aa9732e8ec4a065a7ace45fb0dda9e877b2acb4f42e33b663236ef2dcb WatchSource:0}: Error finding container 3722e4aa9732e8ec4a065a7ace45fb0dda9e877b2acb4f42e33b663236ef2dcb: Status 404 returned error can't find the container with id 3722e4aa9732e8ec4a065a7ace45fb0dda9e877b2acb4f42e33b663236ef2dcb
Jul 28 17:13:45.328687 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:45.328605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh" event={"ID":"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5","Type":"ContainerStarted","Data":"3722e4aa9732e8ec4a065a7ace45fb0dda9e877b2acb4f42e33b663236ef2dcb"}
Jul 28 17:13:47.296737 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:47.296706    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-rdf6p"
Jul 28 17:13:48.345789 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:48.345744    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh" event={"ID":"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5","Type":"ContainerStarted","Data":"794e6d709f8b82d70f23d941832ddc8a5f16e56456fee4aa5f0c1f12a6cd72e2"}
Jul 28 17:13:49.351578 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:49.351542    2560 generic.go:356] "Generic (PLEG): container finished" podID="22627ec9-dc12-4a35-9982-4e0fb1cdc9b5" containerID="794e6d709f8b82d70f23d941832ddc8a5f16e56456fee4aa5f0c1f12a6cd72e2" exitCode=0
Jul 28 17:13:49.351965 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:49.351581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh" event={"ID":"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5","Type":"ContainerDied","Data":"794e6d709f8b82d70f23d941832ddc8a5f16e56456fee4aa5f0c1f12a6cd72e2"}
Jul 28 17:13:49.723497 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:49.723415    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:13:49.726454 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:49.726429    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:13:49.729072 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:49.729052    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:13:50.358263 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:50.358222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh" event={"ID":"22627ec9-dc12-4a35-9982-4e0fb1cdc9b5","Type":"ContainerStarted","Data":"d01d93b27101c0e0668f1d4ff83559bbbdddacce160407fd286e8fd3045d3f8e"}
Jul 28 17:13:50.360263 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:50.358564    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:13:50.394440 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:13:50.394375    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh" podStartSLOduration=1.680164658 podStartE2EDuration="6.39433506s" totalImagesPullingTime="3.625935269s" totalInitContainerRuntime="1.088235133s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:50.392894052 +0000 UTC m=+301.065221974" watchObservedRunningTime="2026-07-28 17:13:50.39433506 +0000 UTC m=+301.066662980"
Jul 28 17:14:01.364978 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:01.364941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh"
Jul 28 17:14:04.490575 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.490535    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"]
Jul 28 17:14:04.590054 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.590012    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"]
Jul 28 17:14:04.590239 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.590059    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.597097 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.597072    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:14:04.597861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.597838    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 17:14:04.597973 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.597839    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 17:14:04.597973 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.597886    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-d5qnv\""
Jul 28 17:14:04.597973 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.597906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:14:04.667556 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.667526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c92759c-0bcf-46ff-8161-74bde87bb80f-tmp\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.667748 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.667580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c92759c-0bcf-46ff-8161-74bde87bb80f-tls-certs\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.667748 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.667676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zx8\" (UniqueName: \"kubernetes.io/projected/7c92759c-0bcf-46ff-8161-74bde87bb80f-kube-api-access-r7zx8\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.768682 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.768569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c92759c-0bcf-46ff-8161-74bde87bb80f-tls-certs\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.768682 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.768656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zx8\" (UniqueName: \"kubernetes.io/projected/7c92759c-0bcf-46ff-8161-74bde87bb80f-kube-api-access-r7zx8\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.768921 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.768710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c92759c-0bcf-46ff-8161-74bde87bb80f-tmp\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.771579 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.771522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c92759c-0bcf-46ff-8161-74bde87bb80f-tls-certs\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.771898 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.771874    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c92759c-0bcf-46ff-8161-74bde87bb80f-tmp\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.790590 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.790562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zx8\" (UniqueName: \"kubernetes.io/projected/7c92759c-0bcf-46ff-8161-74bde87bb80f-kube-api-access-r7zx8\") pod \"kube-auth-proxy-77b678cfbc-f2cz8\" (UID: \"7c92759c-0bcf-46ff-8161-74bde87bb80f\") " pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:04.899781 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:04.899751    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"
Jul 28 17:14:05.058428 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:05.058392    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8"]
Jul 28 17:14:05.061488 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:14:05.061461    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c92759c_0bcf_46ff_8161_74bde87bb80f.slice/crio-6604958c5610012f3cab9abc02fe99390ec4093bcb34d5418a90161ad203e60b WatchSource:0}: Error finding container 6604958c5610012f3cab9abc02fe99390ec4093bcb34d5418a90161ad203e60b: Status 404 returned error can't find the container with id 6604958c5610012f3cab9abc02fe99390ec4093bcb34d5418a90161ad203e60b
Jul 28 17:14:05.063370 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:05.063337    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:14:05.424259 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:05.424171    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8" event={"ID":"7c92759c-0bcf-46ff-8161-74bde87bb80f","Type":"ContainerStarted","Data":"6604958c5610012f3cab9abc02fe99390ec4093bcb34d5418a90161ad203e60b"}
Jul 28 17:14:08.445153 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:08.445115    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8" event={"ID":"7c92759c-0bcf-46ff-8161-74bde87bb80f","Type":"ContainerStarted","Data":"03400b4177824ee13b388666fbfce408c0ece7b682326c62fbb653d3a6ba8666"}
Jul 28 17:14:08.464534 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:14:08.464475    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77b678cfbc-f2cz8" podStartSLOduration=2.00981676 podStartE2EDuration="4.464455367s" totalImagesPullingTime="2.454638607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:08.463667129 +0000 UTC m=+319.135995053" watchObservedRunningTime="2026-07-28 17:14:08.464455367 +0000 UTC m=+319.136783289"
Jul 28 17:15:51.181047 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.181006    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"]
Jul 28 17:15:51.183360 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.183321    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.186160 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.186136    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 17:15:51.186287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.186191    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m4mtg\""
Jul 28 17:15:51.186287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.186137    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 17:15:51.187019 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.186998    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 17:15:51.187118 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.187030    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 17:15:51.193452 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.193431    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"]
Jul 28 17:15:51.294119 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.294067    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5283888-8a98-4238-b681-8e5d241e1e75-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.294371 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.294211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ptt\" (UniqueName: \"kubernetes.io/projected/f5283888-8a98-4238-b681-8e5d241e1e75-kube-api-access-m7ptt\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.294371 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.294286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f5283888-8a98-4238-b681-8e5d241e1e75-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.395156 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.395114    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f5283888-8a98-4238-b681-8e5d241e1e75-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.395339 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.395191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5283888-8a98-4238-b681-8e5d241e1e75-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.395339 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.395265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ptt\" (UniqueName: \"kubernetes.io/projected/f5283888-8a98-4238-b681-8e5d241e1e75-kube-api-access-m7ptt\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.395339 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:15:51.395311    2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 17:15:51.395502 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:15:51.395452    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5283888-8a98-4238-b681-8e5d241e1e75-plugin-serving-cert podName:f5283888-8a98-4238-b681-8e5d241e1e75 nodeName:}" failed. No retries permitted until 2026-07-28 17:15:51.895428883 +0000 UTC m=+422.567756785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f5283888-8a98-4238-b681-8e5d241e1e75-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-4d4hl" (UID: "f5283888-8a98-4238-b681-8e5d241e1e75") : secret "plugin-serving-cert" not found
Jul 28 17:15:51.395880 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.395860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f5283888-8a98-4238-b681-8e5d241e1e75-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.495377 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.495266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ptt\" (UniqueName: \"kubernetes.io/projected/f5283888-8a98-4238-b681-8e5d241e1e75-kube-api-access-m7ptt\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.900117 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.900080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5283888-8a98-4238-b681-8e5d241e1e75-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:51.902484 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:51.902453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5283888-8a98-4238-b681-8e5d241e1e75-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4d4hl\" (UID: \"f5283888-8a98-4238-b681-8e5d241e1e75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:52.093882 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:52.093843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"
Jul 28 17:15:52.226115 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:52.226090    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl"]
Jul 28 17:15:52.229167 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:15:52.229141    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5283888_8a98_4238_b681_8e5d241e1e75.slice/crio-f046d800a5ac8d6276204b1d5213528ce5c6898c74ed935ea161876a3dbe781a WatchSource:0}: Error finding container f046d800a5ac8d6276204b1d5213528ce5c6898c74ed935ea161876a3dbe781a: Status 404 returned error can't find the container with id f046d800a5ac8d6276204b1d5213528ce5c6898c74ed935ea161876a3dbe781a
Jul 28 17:15:52.913510 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:15:52.913472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl" event={"ID":"f5283888-8a98-4238-b681-8e5d241e1e75","Type":"ContainerStarted","Data":"f046d800a5ac8d6276204b1d5213528ce5c6898c74ed935ea161876a3dbe781a"}
Jul 28 17:16:02.401318 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.401286    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"]
Jul 28 17:16:02.406845 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.406583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"
Jul 28 17:16:02.411549 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.411521    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-h58ll\""
Jul 28 17:16:02.417984 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.417957    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"]
Jul 28 17:16:02.500275 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.500232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lm9p\" (UniqueName: \"kubernetes.io/projected/59ed6025-d4d5-41cd-b3b0-c2043f31b462-kube-api-access-9lm9p\") pod \"limitador-operator-controller-manager-859d7b78bd-m6jh5\" (UID: \"59ed6025-d4d5-41cd-b3b0-c2043f31b462\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"
Jul 28 17:16:02.601789 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.601693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lm9p\" (UniqueName: \"kubernetes.io/projected/59ed6025-d4d5-41cd-b3b0-c2043f31b462-kube-api-access-9lm9p\") pod \"limitador-operator-controller-manager-859d7b78bd-m6jh5\" (UID: \"59ed6025-d4d5-41cd-b3b0-c2043f31b462\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"
Jul 28 17:16:02.625539 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.625506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lm9p\" (UniqueName: \"kubernetes.io/projected/59ed6025-d4d5-41cd-b3b0-c2043f31b462-kube-api-access-9lm9p\") pod \"limitador-operator-controller-manager-859d7b78bd-m6jh5\" (UID: \"59ed6025-d4d5-41cd-b3b0-c2043f31b462\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"
Jul 28 17:16:02.719668 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.719576    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"
Jul 28 17:16:02.870577 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.870552    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"]
Jul 28 17:16:02.872493 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:16:02.872456    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ed6025_d4d5_41cd_b3b0_c2043f31b462.slice/crio-cd08b32e32df0983b62e8f7fc46018016b87f508cc795cdb8a6073f3d211cb90 WatchSource:0}: Error finding container cd08b32e32df0983b62e8f7fc46018016b87f508cc795cdb8a6073f3d211cb90: Status 404 returned error can't find the container with id cd08b32e32df0983b62e8f7fc46018016b87f508cc795cdb8a6073f3d211cb90
Jul 28 17:16:02.966461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:02.966427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5" event={"ID":"59ed6025-d4d5-41cd-b3b0-c2043f31b462","Type":"ContainerStarted","Data":"cd08b32e32df0983b62e8f7fc46018016b87f508cc795cdb8a6073f3d211cb90"}
Jul 28 17:16:05.981657 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:05.981619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5" event={"ID":"59ed6025-d4d5-41cd-b3b0-c2043f31b462","Type":"ContainerStarted","Data":"812900d05f5462419711c3e3ed8c4f87bc080360793a28b5b7ef42b744c56b74"}
Jul 28 17:16:05.982108 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:05.981683    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"
Jul 28 17:16:16.988553 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:16.988479    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5"
Jul 28 17:16:17.015696 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:17.015634    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m6jh5" podStartSLOduration=12.603600142 podStartE2EDuration="15.015613006s" totalImagesPullingTime="2.412012864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:06.005768666 +0000 UTC m=+436.678096585" watchObservedRunningTime="2026-07-28 17:16:17.015613006 +0000 UTC m=+447.687940930"
Jul 28 17:16:27.090179 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:27.090098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl" event={"ID":"f5283888-8a98-4238-b681-8e5d241e1e75","Type":"ContainerStarted","Data":"c38ad586cf37fbfd431c2e22a69c81696bc78184242a73c6e98a20ed02ba6008"}
Jul 28 17:16:27.133976 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:16:27.133922    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4d4hl" podStartSLOduration=1.52498266 podStartE2EDuration="36.133903421s" totalImagesPullingTime="34.608920761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:27.131911509 +0000 UTC m=+457.804239431" watchObservedRunningTime="2026-07-28 17:16:27.133903421 +0000 UTC m=+457.806231342"
Jul 28 17:17:58.309613 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.309580    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-msj2v"]
Jul 28 17:17:58.312732 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.312707    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:17:58.353130 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.353099    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 17:17:58.354515 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.354498    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-d8bl4\""
Jul 28 17:17:58.395149 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.395104    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-msj2v"]
Jul 28 17:17:58.460304 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.460267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09869388-033a-467f-ab36-284134b764c2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-msj2v\" (UID: \"09869388-033a-467f-ab36-284134b764c2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:17:58.460502 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.460323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddjg\" (UniqueName: \"kubernetes.io/projected/09869388-033a-467f-ab36-284134b764c2-kube-api-access-lddjg\") pod \"payload-processing-5cc6cd444b-msj2v\" (UID: \"09869388-033a-467f-ab36-284134b764c2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:17:58.561021 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.560908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddjg\" (UniqueName: \"kubernetes.io/projected/09869388-033a-467f-ab36-284134b764c2-kube-api-access-lddjg\") pod \"payload-processing-5cc6cd444b-msj2v\" (UID: \"09869388-033a-467f-ab36-284134b764c2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:17:58.561231 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.561046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09869388-033a-467f-ab36-284134b764c2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-msj2v\" (UID: \"09869388-033a-467f-ab36-284134b764c2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:17:58.561920 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.561894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09869388-033a-467f-ab36-284134b764c2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-msj2v\" (UID: \"09869388-033a-467f-ab36-284134b764c2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:17:58.576041 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.576011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddjg\" (UniqueName: \"kubernetes.io/projected/09869388-033a-467f-ab36-284134b764c2-kube-api-access-lddjg\") pod \"payload-processing-5cc6cd444b-msj2v\" (UID: \"09869388-033a-467f-ab36-284134b764c2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:17:58.622752 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.622708    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:17:58.766198 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:58.766163    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-msj2v"]
Jul 28 17:17:58.774878 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:17:58.774844    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09869388_033a_467f_ab36_284134b764c2.slice/crio-d12344818eff701bf87d4942be269ac4e50617b289bcd21197a540c5c1993df6 WatchSource:0}: Error finding container d12344818eff701bf87d4942be269ac4e50617b289bcd21197a540c5c1993df6: Status 404 returned error can't find the container with id d12344818eff701bf87d4942be269ac4e50617b289bcd21197a540c5c1993df6
Jul 28 17:17:59.516528 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:17:59.516444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v" event={"ID":"09869388-033a-467f-ab36-284134b764c2","Type":"ContainerStarted","Data":"d12344818eff701bf87d4942be269ac4e50617b289bcd21197a540c5c1993df6"}
Jul 28 17:18:01.529077 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:01.529035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v" event={"ID":"09869388-033a-467f-ab36-284134b764c2","Type":"ContainerStarted","Data":"dea86f161bed9e4e197697bee23cb3347e7afd13214781a8b369785e803c048b"}
Jul 28 17:18:01.529548 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:01.529267    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:18:01.584004 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:01.583945    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v" podStartSLOduration=1.876378913 podStartE2EDuration="3.583926521s" totalImagesPullingTime="1.707547608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:01.575548403 +0000 UTC m=+552.247876326" watchObservedRunningTime="2026-07-28 17:18:01.583926521 +0000 UTC m=+552.256254440"
Jul 28 17:18:12.536529 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:12.536492    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-msj2v"
Jul 28 17:18:36.618132 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.618090    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"]
Jul 28 17:18:36.622607 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.622575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.628434 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.628409    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 17:18:36.629364 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.629318    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jddxn\""
Jul 28 17:18:36.629364 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.629331    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 17:18:36.629552 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.629404    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 17:18:36.641969 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.638302    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"]
Jul 28 17:18:36.715017 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.714975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.715198 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.715029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.715198 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.715090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hgv\" (UniqueName: \"kubernetes.io/projected/1570c290-25a7-4e7b-840c-86ea9e54b349-kube-api-access-w6hgv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.715198 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.715135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.715198 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.715156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.715198 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.715176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1570c290-25a7-4e7b-840c-86ea9e54b349-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.816844 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.816802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.817020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.816861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.817020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.816890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hgv\" (UniqueName: \"kubernetes.io/projected/1570c290-25a7-4e7b-840c-86ea9e54b349-kube-api-access-w6hgv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.817020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.816914    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.817020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.816941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.817020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.816970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1570c290-25a7-4e7b-840c-86ea9e54b349-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.817288 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.817257    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.817403 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.817318    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.817455 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.817412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.819326 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.819304    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1570c290-25a7-4e7b-840c-86ea9e54b349-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.819574 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.819558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1570c290-25a7-4e7b-840c-86ea9e54b349-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.837984 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.837952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hgv\" (UniqueName: \"kubernetes.io/projected/1570c290-25a7-4e7b-840c-86ea9e54b349-kube-api-access-w6hgv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkljt\" (UID: \"1570c290-25a7-4e7b-840c-86ea9e54b349\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:36.933118 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:36.933014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:37.080508 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:37.080474    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"]
Jul 28 17:18:37.084503 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:18:37.084471    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1570c290_25a7_4e7b_840c_86ea9e54b349.slice/crio-47a30827e5b4361980cf5a0227be8f588b6c3cbe19a799b107f0435986403d5f WatchSource:0}: Error finding container 47a30827e5b4361980cf5a0227be8f588b6c3cbe19a799b107f0435986403d5f: Status 404 returned error can't find the container with id 47a30827e5b4361980cf5a0227be8f588b6c3cbe19a799b107f0435986403d5f
Jul 28 17:18:37.712563 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:37.712514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt" event={"ID":"1570c290-25a7-4e7b-840c-86ea9e54b349","Type":"ContainerStarted","Data":"47a30827e5b4361980cf5a0227be8f588b6c3cbe19a799b107f0435986403d5f"}
Jul 28 17:18:38.718493 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:38.718457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt" event={"ID":"1570c290-25a7-4e7b-840c-86ea9e54b349","Type":"ContainerStarted","Data":"df9743e1c7c015ddd12a69bf423622e9e4b23e89f08346661797438580ad2fa4"}
Jul 28 17:18:38.800355 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:38.800301    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt" podStartSLOduration=1.7167241359999998 podStartE2EDuration="2.80028132s" totalImagesPullingTime="1.083557184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:38.799535586 +0000 UTC m=+589.471863506" watchObservedRunningTime="2026-07-28 17:18:38.80028132 +0000 UTC m=+589.472609240"
Jul 28 17:18:45.464833 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.464798    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"]
Jul 28 17:18:45.469002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.468974    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.472440 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.472412    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 28 17:18:45.497691 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.497659    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"]
Jul 28 17:18:45.597619 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.597569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwh6\" (UniqueName: \"kubernetes.io/projected/d90c89c3-4171-497b-b9f6-1267a3ffb131-kube-api-access-smwh6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.597821 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.597638    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.597821 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.597727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.597821 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.597756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.597821 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.597784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.597821 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.597810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90c89c3-4171-497b-b9f6-1267a3ffb131-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699238 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90c89c3-4171-497b-b9f6-1267a3ffb131-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwh6\" (UniqueName: \"kubernetes.io/projected/d90c89c3-4171-497b-b9f6-1267a3ffb131-kube-api-access-smwh6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699753 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699811 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.699811 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.699779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.701713 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.701692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90c89c3-4171-497b-b9f6-1267a3ffb131-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.701878 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.701860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90c89c3-4171-497b-b9f6-1267a3ffb131-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.712207 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.712178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwh6\" (UniqueName: \"kubernetes.io/projected/d90c89c3-4171-497b-b9f6-1267a3ffb131-kube-api-access-smwh6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-whst8\" (UID: \"d90c89c3-4171-497b-b9f6-1267a3ffb131\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.780338 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.780235    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:45.938690 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:45.938604    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"]
Jul 28 17:18:45.941671 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:18:45.941639    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90c89c3_4171_497b_b9f6_1267a3ffb131.slice/crio-b084d448fa7e862b57e1a8af5f34fa8eef35de3dbd9e37d231b1209031e29dce WatchSource:0}: Error finding container b084d448fa7e862b57e1a8af5f34fa8eef35de3dbd9e37d231b1209031e29dce: Status 404 returned error can't find the container with id b084d448fa7e862b57e1a8af5f34fa8eef35de3dbd9e37d231b1209031e29dce
Jul 28 17:18:46.756834 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:46.756797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8" event={"ID":"d90c89c3-4171-497b-b9f6-1267a3ffb131","Type":"ContainerStarted","Data":"0e3af49ddf6cc48535bc9a04e6befd3fdf44dd13b1d4083cfd993feb7e996eab"}
Jul 28 17:18:46.757307 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:46.756852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8" event={"ID":"d90c89c3-4171-497b-b9f6-1267a3ffb131","Type":"ContainerStarted","Data":"b084d448fa7e862b57e1a8af5f34fa8eef35de3dbd9e37d231b1209031e29dce"}
Jul 28 17:18:46.786538 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:46.786481    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8" podStartSLOduration=1.589862163 podStartE2EDuration="1.786464166s" totalImagesPullingTime="196.602003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:46.785708105 +0000 UTC m=+597.458036083" watchObservedRunningTime="2026-07-28 17:18:46.786464166 +0000 UTC m=+597.458792113"
Jul 28 17:18:46.933166 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:46.933122    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:46.933396 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:46.933215    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:46.945697 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:46.945670    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:47.772808 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:47.772776    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkljt"
Jul 28 17:18:49.755628 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:49.755596    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:18:49.758542 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:49.758519    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:18:55.781267 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:55.781225    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:55.783977 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:55.781289    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:55.793627 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:55.793607    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:18:55.809092 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:18:55.809072    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-whst8"
Jul 28 17:23:49.786578 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:23:49.786554    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:23:49.787435 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:23:49.787415    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:28:49.815408 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:28:49.815324    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:28:49.817495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:28:49.817474    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:30:27.711536 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.711502    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"]
Jul 28 17:30:27.714920 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.714903    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.718132 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.718112    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-01e6fde9-kserve-self-signed-certs\""
Jul 28 17:30:27.730087 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.730056    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"]
Jul 28 17:30:27.807817 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.807778    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-tmp-dir\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.807817 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.807818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff52183-70cf-4636-8a08-a46bb60c29d1-tls-certs\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.808040 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.807892    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7xm\" (UniqueName: \"kubernetes.io/projected/eff52183-70cf-4636-8a08-a46bb60c29d1-kube-api-access-ch7xm\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.808040 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.807937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-dshm\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.808040 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.807962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-home\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.808195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.808107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-model-cache\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909080 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-home\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909281 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-model-cache\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909281 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-tmp-dir\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909281 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff52183-70cf-4636-8a08-a46bb60c29d1-tls-certs\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909281 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7xm\" (UniqueName: \"kubernetes.io/projected/eff52183-70cf-4636-8a08-a46bb60c29d1-kube-api-access-ch7xm\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909281 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-dshm\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909578 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-home\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-tmp-dir\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.909658 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.909616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-model-cache\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.911454 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.911433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-dshm\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.911627 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.911610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff52183-70cf-4636-8a08-a46bb60c29d1-tls-certs\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:27.919159 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:27.919134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7xm\" (UniqueName: \"kubernetes.io/projected/eff52183-70cf-4636-8a08-a46bb60c29d1-kube-api-access-ch7xm\") pod \"e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") " pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:28.023583 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:28.023498    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:28.163260 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:28.163237    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"]
Jul 28 17:30:28.934189 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:28.934151    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t" event={"ID":"eff52183-70cf-4636-8a08-a46bb60c29d1","Type":"ContainerStarted","Data":"87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453"}
Jul 28 17:30:28.934582 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:28.934198    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t" event={"ID":"eff52183-70cf-4636-8a08-a46bb60c29d1","Type":"ContainerStarted","Data":"cf44610d18f292948444ffcda2caf8c5512546c1e0c511d9ebe4638ac0a6a8ad"}
Jul 28 17:30:28.956678 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:28.956631    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t" podStartSLOduration=1.956618191 podStartE2EDuration="1.956618191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:28.955529427 +0000 UTC m=+1299.627857349" watchObservedRunningTime="2026-07-28 17:30:28.956618191 +0000 UTC m=+1299.628946110"
Jul 28 17:30:38.023863 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:38.023832    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:38.024219 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:38.023879    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:38.036469 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:38.036440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:38.990157 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:38.990126    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:30:46.591000 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.590970    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"]
Jul 28 17:30:46.597417 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.595400    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.598721 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.598689    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-01e6fde9-kserve-self-signed-certs\""
Jul 28 17:30:46.603890 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.603868    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"]
Jul 28 17:30:46.680789 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.680767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-dshm\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.680904 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.680795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-tmp-dir\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.680904 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.680831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-home\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.680988 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.680912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-model-cache\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.680988 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.680962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28eeefec-8c80-417d-b58b-293000a27470-tls-certs\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.681060 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.680994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwvs\" (UniqueName: \"kubernetes.io/projected/28eeefec-8c80-417d-b58b-293000a27470-kube-api-access-bkwvs\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781438 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-home\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781561 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-model-cache\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781561 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28eeefec-8c80-417d-b58b-293000a27470-tls-certs\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781561 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwvs\" (UniqueName: \"kubernetes.io/projected/28eeefec-8c80-417d-b58b-293000a27470-kube-api-access-bkwvs\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781709 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-dshm\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781709 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-tmp-dir\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781864 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-model-cache\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781864 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-home\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.781989 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.781969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-tmp-dir\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.783727 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.783708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-dshm\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.783912 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.783895    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28eeefec-8c80-417d-b58b-293000a27470-tls-certs\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.790447 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.790429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwvs\" (UniqueName: \"kubernetes.io/projected/28eeefec-8c80-417d-b58b-293000a27470-kube-api-access-bkwvs\") pod \"e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") " pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:46.907521 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:46.907456    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:47.041983 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:47.041961    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"]
Jul 28 17:30:47.043713 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:30:47.043684    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28eeefec_8c80_417d_b58b_293000a27470.slice/crio-61646ff72d7cbb60228499e4a2bc6a043ba65c8cbbe4b8b3656f3e9f3eeee7dd WatchSource:0}: Error finding container 61646ff72d7cbb60228499e4a2bc6a043ba65c8cbbe4b8b3656f3e9f3eeee7dd: Status 404 returned error can't find the container with id 61646ff72d7cbb60228499e4a2bc6a043ba65c8cbbe4b8b3656f3e9f3eeee7dd
Jul 28 17:30:48.021596 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:48.021564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6" event={"ID":"28eeefec-8c80-417d-b58b-293000a27470","Type":"ContainerStarted","Data":"3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73"}
Jul 28 17:30:48.021596 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:48.021602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6" event={"ID":"28eeefec-8c80-417d-b58b-293000a27470","Type":"ContainerStarted","Data":"61646ff72d7cbb60228499e4a2bc6a043ba65c8cbbe4b8b3656f3e9f3eeee7dd"}
Jul 28 17:30:48.041224 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:48.041180    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6" podStartSLOduration=2.041164564 podStartE2EDuration="2.041164564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:48.03852708 +0000 UTC m=+1318.710854979" watchObservedRunningTime="2026-07-28 17:30:48.041164564 +0000 UTC m=+1318.713492484"
Jul 28 17:30:56.908142 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:56.908099    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:56.908142 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:56.908149    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:56.920708 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:56.920686    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:30:57.078470 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:30:57.078439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:31:06.663144 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:06.663109    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"]
Jul 28 17:31:08.242487 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:08.242456    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"]
Jul 28 17:31:21.681992 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:21.681899    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t" podUID="eff52183-70cf-4636-8a08-a46bb60c29d1" containerName="main" containerID="cri-o://87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453" gracePeriod=45
Jul 28 17:31:21.929082 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:21.929058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:31:22.110646 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.110600    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-home\") pod \"eff52183-70cf-4636-8a08-a46bb60c29d1\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") "
Jul 28 17:31:22.110842 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.110667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-tmp-dir\") pod \"eff52183-70cf-4636-8a08-a46bb60c29d1\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") "
Jul 28 17:31:22.110842 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.110686    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch7xm\" (UniqueName: \"kubernetes.io/projected/eff52183-70cf-4636-8a08-a46bb60c29d1-kube-api-access-ch7xm\") pod \"eff52183-70cf-4636-8a08-a46bb60c29d1\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") "
Jul 28 17:31:22.110842 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.110713    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff52183-70cf-4636-8a08-a46bb60c29d1-tls-certs\") pod \"eff52183-70cf-4636-8a08-a46bb60c29d1\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") "
Jul 28 17:31:22.110842 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.110738    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-dshm\") pod \"eff52183-70cf-4636-8a08-a46bb60c29d1\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") "
Jul 28 17:31:22.110842 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.110824    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-model-cache\") pod \"eff52183-70cf-4636-8a08-a46bb60c29d1\" (UID: \"eff52183-70cf-4636-8a08-a46bb60c29d1\") "
Jul 28 17:31:22.111103 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.110880    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-home" (OuterVolumeSpecName: "home") pod "eff52183-70cf-4636-8a08-a46bb60c29d1" (UID: "eff52183-70cf-4636-8a08-a46bb60c29d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:22.111103 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.111019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eff52183-70cf-4636-8a08-a46bb60c29d1" (UID: "eff52183-70cf-4636-8a08-a46bb60c29d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:22.111103 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.111046    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "eff52183-70cf-4636-8a08-a46bb60c29d1" (UID: "eff52183-70cf-4636-8a08-a46bb60c29d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:22.111279 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.111191    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:22.111279 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.111209    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:22.111279 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.111223    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:22.112958 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.112938    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-dshm" (OuterVolumeSpecName: "dshm") pod "eff52183-70cf-4636-8a08-a46bb60c29d1" (UID: "eff52183-70cf-4636-8a08-a46bb60c29d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:22.113038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.113015    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff52183-70cf-4636-8a08-a46bb60c29d1-kube-api-access-ch7xm" (OuterVolumeSpecName: "kube-api-access-ch7xm") pod "eff52183-70cf-4636-8a08-a46bb60c29d1" (UID: "eff52183-70cf-4636-8a08-a46bb60c29d1"). InnerVolumeSpecName "kube-api-access-ch7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:22.113082 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.113049    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff52183-70cf-4636-8a08-a46bb60c29d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eff52183-70cf-4636-8a08-a46bb60c29d1" (UID: "eff52183-70cf-4636-8a08-a46bb60c29d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:31:22.184802 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.184771    2560 generic.go:356] "Generic (PLEG): container finished" podID="eff52183-70cf-4636-8a08-a46bb60c29d1" containerID="87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453" exitCode=0
Jul 28 17:31:22.184958 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.184814    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"
Jul 28 17:31:22.184958 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.184852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t" event={"ID":"eff52183-70cf-4636-8a08-a46bb60c29d1","Type":"ContainerDied","Data":"87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453"}
Jul 28 17:31:22.184958 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.184895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t" event={"ID":"eff52183-70cf-4636-8a08-a46bb60c29d1","Type":"ContainerDied","Data":"cf44610d18f292948444ffcda2caf8c5512546c1e0c511d9ebe4638ac0a6a8ad"}
Jul 28 17:31:22.184958 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.184912    2560 scope.go:117] "RemoveContainer" containerID="87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453"
Jul 28 17:31:22.198796 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.198769    2560 scope.go:117] "RemoveContainer" containerID="87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453"
Jul 28 17:31:22.199085 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:31:22.199065    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453\": container with ID starting with 87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453 not found: ID does not exist" containerID="87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453"
Jul 28 17:31:22.199138 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.199094    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453"} err="failed to get container status \"87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453\": rpc error: code = NotFound desc = could not find container \"87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453\": container with ID starting with 87a5d80a140189f23f27977b048b7ce19bd742e1f6e118be9158bf4b578d3453 not found: ID does not exist"
Jul 28 17:31:22.212375 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.212138    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff52183-70cf-4636-8a08-a46bb60c29d1-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:22.212375 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.212163    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7xm\" (UniqueName: \"kubernetes.io/projected/eff52183-70cf-4636-8a08-a46bb60c29d1-kube-api-access-ch7xm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:22.212375 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.212178    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff52183-70cf-4636-8a08-a46bb60c29d1-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:22.214248 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.214228    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"]
Jul 28 17:31:22.219166 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:22.219146    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-01e6fde9-kserve-764f788999-gq52t"]
Jul 28 17:31:23.260809 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.260747    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6" podUID="28eeefec-8c80-417d-b58b-293000a27470" containerName="main" containerID="cri-o://3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73" gracePeriod=45
Jul 28 17:31:23.511806 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.511749    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:31:23.524251 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524224    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-dshm\") pod \"28eeefec-8c80-417d-b58b-293000a27470\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") "
Jul 28 17:31:23.524378 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524258    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28eeefec-8c80-417d-b58b-293000a27470-tls-certs\") pod \"28eeefec-8c80-417d-b58b-293000a27470\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") "
Jul 28 17:31:23.524378 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524285    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-model-cache\") pod \"28eeefec-8c80-417d-b58b-293000a27470\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") "
Jul 28 17:31:23.524378 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524301    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkwvs\" (UniqueName: \"kubernetes.io/projected/28eeefec-8c80-417d-b58b-293000a27470-kube-api-access-bkwvs\") pod \"28eeefec-8c80-417d-b58b-293000a27470\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") "
Jul 28 17:31:23.524378 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524320    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-tmp-dir\") pod \"28eeefec-8c80-417d-b58b-293000a27470\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") "
Jul 28 17:31:23.524378 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524339    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-home\") pod \"28eeefec-8c80-417d-b58b-293000a27470\" (UID: \"28eeefec-8c80-417d-b58b-293000a27470\") "
Jul 28 17:31:23.524636 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524568    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-model-cache" (OuterVolumeSpecName: "model-cache") pod "28eeefec-8c80-417d-b58b-293000a27470" (UID: "28eeefec-8c80-417d-b58b-293000a27470"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:23.524636 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524581    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28eeefec-8c80-417d-b58b-293000a27470" (UID: "28eeefec-8c80-417d-b58b-293000a27470"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:23.524712 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.524658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-home" (OuterVolumeSpecName: "home") pod "28eeefec-8c80-417d-b58b-293000a27470" (UID: "28eeefec-8c80-417d-b58b-293000a27470"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:23.526387 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.526363    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-dshm" (OuterVolumeSpecName: "dshm") pod "28eeefec-8c80-417d-b58b-293000a27470" (UID: "28eeefec-8c80-417d-b58b-293000a27470"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:23.526473 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.526365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28eeefec-8c80-417d-b58b-293000a27470-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28eeefec-8c80-417d-b58b-293000a27470" (UID: "28eeefec-8c80-417d-b58b-293000a27470"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:31:23.526473 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.526399    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28eeefec-8c80-417d-b58b-293000a27470-kube-api-access-bkwvs" (OuterVolumeSpecName: "kube-api-access-bkwvs") pod "28eeefec-8c80-417d-b58b-293000a27470" (UID: "28eeefec-8c80-417d-b58b-293000a27470"). InnerVolumeSpecName "kube-api-access-bkwvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:23.625836 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.625807    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:23.625836 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.625831    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28eeefec-8c80-417d-b58b-293000a27470-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:23.625836 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.625842    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:23.626037 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.625850    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkwvs\" (UniqueName: \"kubernetes.io/projected/28eeefec-8c80-417d-b58b-293000a27470-kube-api-access-bkwvs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:23.626037 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.625858    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:23.626037 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.625866    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28eeefec-8c80-417d-b58b-293000a27470-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:31:23.872610 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:23.872568    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff52183-70cf-4636-8a08-a46bb60c29d1" path="/var/lib/kubelet/pods/eff52183-70cf-4636-8a08-a46bb60c29d1/volumes"
Jul 28 17:31:24.196773 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.196672    2560 generic.go:356] "Generic (PLEG): container finished" podID="28eeefec-8c80-417d-b58b-293000a27470" containerID="3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73" exitCode=0
Jul 28 17:31:24.196773 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.196729    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"
Jul 28 17:31:24.196773 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.196759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6" event={"ID":"28eeefec-8c80-417d-b58b-293000a27470","Type":"ContainerDied","Data":"3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73"}
Jul 28 17:31:24.197061 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.196802    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6" event={"ID":"28eeefec-8c80-417d-b58b-293000a27470","Type":"ContainerDied","Data":"61646ff72d7cbb60228499e4a2bc6a043ba65c8cbbe4b8b3656f3e9f3eeee7dd"}
Jul 28 17:31:24.197061 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.196822    2560 scope.go:117] "RemoveContainer" containerID="3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73"
Jul 28 17:31:24.205224 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.205206    2560 scope.go:117] "RemoveContainer" containerID="3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73"
Jul 28 17:31:24.205499 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:31:24.205480    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73\": container with ID starting with 3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73 not found: ID does not exist" containerID="3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73"
Jul 28 17:31:24.205581 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.205508    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73"} err="failed to get container status \"3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73\": rpc error: code = NotFound desc = could not find container \"3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73\": container with ID starting with 3c7d3c47b3f0d452af77e4f5bcac998068c62d93e6829d139fd1b031f9a79b73 not found: ID does not exist"
Jul 28 17:31:24.220849 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.220828    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"]
Jul 28 17:31:24.227333 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:24.227314    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-01e6fde9-kserve-5d6d9dbcbc-zbhx6"]
Jul 28 17:31:25.872681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:31:25.872639    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28eeefec-8c80-417d-b58b-293000a27470" path="/var/lib/kubelet/pods/28eeefec-8c80-417d-b58b-293000a27470/volumes"
Jul 28 17:33:49.843708 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:33:49.843679    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:33:49.846276 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:33:49.846254    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:34:55.900407 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.900369    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"]
Jul 28 17:34:55.900952 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.900709    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff52183-70cf-4636-8a08-a46bb60c29d1" containerName="main"
Jul 28 17:34:55.900952 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.900719    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff52183-70cf-4636-8a08-a46bb60c29d1" containerName="main"
Jul 28 17:34:55.900952 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.900730    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28eeefec-8c80-417d-b58b-293000a27470" containerName="main"
Jul 28 17:34:55.900952 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.900736    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="28eeefec-8c80-417d-b58b-293000a27470" containerName="main"
Jul 28 17:34:55.900952 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.900803    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff52183-70cf-4636-8a08-a46bb60c29d1" containerName="main"
Jul 28 17:34:55.900952 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.900813    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="28eeefec-8c80-417d-b58b-293000a27470" containerName="main"
Jul 28 17:34:55.903899 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.903880    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:55.906442 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.906416    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-949b0a76-serving-cert\""
Jul 28 17:34:55.906543 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.906467    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:34:55.906543 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.906484    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:34:55.907138 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.907119    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:34:55.916110 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.916085    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"]
Jul 28 17:34:55.996844 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.996817    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72caeaf5-3633-4e49-8091-4eae1faddbc2-openshift-service-ca\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:55.996950 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.996862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:55.997309 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:55.997281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8q69\" (UniqueName: \"kubernetes.io/projected/72caeaf5-3633-4e49-8091-4eae1faddbc2-kube-api-access-x8q69\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.098059 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.098033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8q69\" (UniqueName: \"kubernetes.io/projected/72caeaf5-3633-4e49-8091-4eae1faddbc2-kube-api-access-x8q69\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.098164 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.098082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72caeaf5-3633-4e49-8091-4eae1faddbc2-openshift-service-ca\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.098164 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.098111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.098280 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:34:56.098269    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-serving-cert: secret "maas-api-e2e-ait-949b0a76-serving-cert" not found
Jul 28 17:34:56.098328 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:34:56.098317    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls podName:72caeaf5-3633-4e49-8091-4eae1faddbc2 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:56.598302372 +0000 UTC m=+1567.270630270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls") pod "maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5" (UID: "72caeaf5-3633-4e49-8091-4eae1faddbc2") : secret "maas-api-e2e-ait-949b0a76-serving-cert" not found
Jul 28 17:34:56.098718 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.098699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72caeaf5-3633-4e49-8091-4eae1faddbc2-openshift-service-ca\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.106846 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.106827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8q69\" (UniqueName: \"kubernetes.io/projected/72caeaf5-3633-4e49-8091-4eae1faddbc2-kube-api-access-x8q69\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.602387 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.602335    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.604649 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.604627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls\") pod \"maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.738676 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.738644    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"]
Jul 28 17:34:56.742269 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.742254    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:56.744540 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.744518    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-949b0a76\""
Jul 28 17:34:56.744664 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.744571    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-949b0a76-dockercfg-hdvw8\""
Jul 28 17:34:56.753232 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.753208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"]
Jul 28 17:34:56.813893 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.813872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:34:56.881430 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.881402    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"]
Jul 28 17:34:56.885730 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.885707    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:56.901070 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.901047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"]
Jul 28 17:34:56.904993 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.904970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfec42e3-7c2a-4e36-aa77-6b789c055a93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk\" (UID: \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:56.905105 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.905040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wzm\" (UniqueName: \"kubernetes.io/projected/dfec42e3-7c2a-4e36-aa77-6b789c055a93-kube-api-access-s6wzm\") pod \"payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk\" (UID: \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:56.951445 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.951425    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"]
Jul 28 17:34:56.954193 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:34:56.954171    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72caeaf5_3633_4e49_8091_4eae1faddbc2.slice/crio-64d40b126e21d4109cf73daae4ecb2d5f475ac9890b2385ca33f4769499eddb0 WatchSource:0}: Error finding container 64d40b126e21d4109cf73daae4ecb2d5f475ac9890b2385ca33f4769499eddb0: Status 404 returned error can't find the container with id 64d40b126e21d4109cf73daae4ecb2d5f475ac9890b2385ca33f4769499eddb0
Jul 28 17:34:56.955495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:56.955477    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:34:57.005908 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.005887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wzm\" (UniqueName: \"kubernetes.io/projected/dfec42e3-7c2a-4e36-aa77-6b789c055a93-kube-api-access-s6wzm\") pod \"payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk\" (UID: \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:57.006018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.005928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238652fd-f3a2-4ee9-9b64-533b62078fea-plugins-config-volume\") pod \"payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2\" (UID: \"238652fd-f3a2-4ee9-9b64-533b62078fea\") " pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:57.006018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.005955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqd9s\" (UniqueName: \"kubernetes.io/projected/238652fd-f3a2-4ee9-9b64-533b62078fea-kube-api-access-rqd9s\") pod \"payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2\" (UID: \"238652fd-f3a2-4ee9-9b64-533b62078fea\") " pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:57.006272 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.006054    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfec42e3-7c2a-4e36-aa77-6b789c055a93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk\" (UID: \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:57.006857 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.006834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfec42e3-7c2a-4e36-aa77-6b789c055a93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk\" (UID: \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:57.015499 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.015476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wzm\" (UniqueName: \"kubernetes.io/projected/dfec42e3-7c2a-4e36-aa77-6b789c055a93-kube-api-access-s6wzm\") pod \"payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk\" (UID: \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:57.052926 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.052907    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:57.107197 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.107075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238652fd-f3a2-4ee9-9b64-533b62078fea-plugins-config-volume\") pod \"payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2\" (UID: \"238652fd-f3a2-4ee9-9b64-533b62078fea\") " pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:57.107197 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.107120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqd9s\" (UniqueName: \"kubernetes.io/projected/238652fd-f3a2-4ee9-9b64-533b62078fea-kube-api-access-rqd9s\") pod \"payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2\" (UID: \"238652fd-f3a2-4ee9-9b64-533b62078fea\") " pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:57.108438 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.108416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238652fd-f3a2-4ee9-9b64-533b62078fea-plugins-config-volume\") pod \"payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2\" (UID: \"238652fd-f3a2-4ee9-9b64-533b62078fea\") " pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:57.116100 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.116041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqd9s\" (UniqueName: \"kubernetes.io/projected/238652fd-f3a2-4ee9-9b64-533b62078fea-kube-api-access-rqd9s\") pod \"payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2\" (UID: \"238652fd-f3a2-4ee9-9b64-533b62078fea\") " pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:57.167193 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.167164    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5" event={"ID":"72caeaf5-3633-4e49-8091-4eae1faddbc2","Type":"ContainerStarted","Data":"64d40b126e21d4109cf73daae4ecb2d5f475ac9890b2385ca33f4769499eddb0"}
Jul 28 17:34:57.186896 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.186816    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"]
Jul 28 17:34:57.190563 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:34:57.190531    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfec42e3_7c2a_4e36_aa77_6b789c055a93.slice/crio-9de49939929f8a0a31dd26aec29f6e45c382f679b60ee681aa95ae80b2d12350 WatchSource:0}: Error finding container 9de49939929f8a0a31dd26aec29f6e45c382f679b60ee681aa95ae80b2d12350: Status 404 returned error can't find the container with id 9de49939929f8a0a31dd26aec29f6e45c382f679b60ee681aa95ae80b2d12350
Jul 28 17:34:57.196420 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.196381    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:57.333069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:57.332970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"]
Jul 28 17:34:57.335294 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:34:57.335271    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238652fd_f3a2_4ee9_9b64_533b62078fea.slice/crio-17ca4e565a82eae8200a347ae969632c6840858ab33708f4aef56c13cac5e0a2 WatchSource:0}: Error finding container 17ca4e565a82eae8200a347ae969632c6840858ab33708f4aef56c13cac5e0a2: Status 404 returned error can't find the container with id 17ca4e565a82eae8200a347ae969632c6840858ab33708f4aef56c13cac5e0a2
Jul 28 17:34:58.175965 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:58.175726    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk" event={"ID":"dfec42e3-7c2a-4e36-aa77-6b789c055a93","Type":"ContainerStarted","Data":"7169211bdaeb99fe4f225aea26895d0f6251cc210afda5b9355f6eaa164e0384"}
Jul 28 17:34:58.175965 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:58.175907    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:34:58.175965 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:58.175933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk" event={"ID":"dfec42e3-7c2a-4e36-aa77-6b789c055a93","Type":"ContainerStarted","Data":"9de49939929f8a0a31dd26aec29f6e45c382f679b60ee681aa95ae80b2d12350"}
Jul 28 17:34:58.177387 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:58.177362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2" event={"ID":"238652fd-f3a2-4ee9-9b64-533b62078fea","Type":"ContainerStarted","Data":"4b54bd3c643b5a49acf880c5ad7b156538fc8b89f99fa12cfe665db25886b122"}
Jul 28 17:34:58.177507 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:58.177411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2" event={"ID":"238652fd-f3a2-4ee9-9b64-533b62078fea","Type":"ContainerStarted","Data":"17ca4e565a82eae8200a347ae969632c6840858ab33708f4aef56c13cac5e0a2"}
Jul 28 17:34:58.177507 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:58.177499    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:34:58.195100 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:58.195061    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk" podStartSLOduration=2.1950483849999998 podStartE2EDuration="2.195048385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:58.193685665 +0000 UTC m=+1568.866013585" watchObservedRunningTime="2026-07-28 17:34:58.195048385 +0000 UTC m=+1568.867376304"
Jul 28 17:34:58.217990 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:34:58.217939    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2" podStartSLOduration=2.217919573 podStartE2EDuration="2.217919573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:58.215489491 +0000 UTC m=+1568.887817413" watchObservedRunningTime="2026-07-28 17:34:58.217919573 +0000 UTC m=+1568.890247493"
Jul 28 17:35:00.192751 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:00.192715    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5" event={"ID":"72caeaf5-3633-4e49-8091-4eae1faddbc2","Type":"ContainerStarted","Data":"da01080f59497260be681c7d244677ea637f6b2632d7d19575a108a1c79a6935"}
Jul 28 17:35:00.193168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:00.192896    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:35:00.215873 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:00.215829    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5" podStartSLOduration=2.935956984 podStartE2EDuration="5.215816862s" totalImagesPullingTime="2.279859878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:00.213015475 +0000 UTC m=+1570.885343398" watchObservedRunningTime="2026-07-28 17:35:00.215816862 +0000 UTC m=+1570.888144778"
Jul 28 17:35:06.203172 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:06.203143    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:35:09.184451 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:09.184420    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:35:09.184817 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:09.184491    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:35:32.103035 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.102999    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"]
Jul 28 17:35:32.103530 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.103194    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5" podUID="72caeaf5-3633-4e49-8091-4eae1faddbc2" containerName="maas-api" containerID="cri-o://da01080f59497260be681c7d244677ea637f6b2632d7d19575a108a1c79a6935" gracePeriod=30
Jul 28 17:35:32.185847 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.185815    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"]
Jul 28 17:35:32.186123 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.186067    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2" podUID="238652fd-f3a2-4ee9-9b64-533b62078fea" containerName="payload-processing" containerID="cri-o://4b54bd3c643b5a49acf880c5ad7b156538fc8b89f99fa12cfe665db25886b122" gracePeriod=30
Jul 28 17:35:32.200625 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.200598    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"]
Jul 28 17:35:32.201221 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.200890    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk" podUID="dfec42e3-7c2a-4e36-aa77-6b789c055a93" containerName="payload-pre-processing" containerID="cri-o://7169211bdaeb99fe4f225aea26895d0f6251cc210afda5b9355f6eaa164e0384" gracePeriod=30
Jul 28 17:35:32.353020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.352634    2560 generic.go:356] "Generic (PLEG): container finished" podID="72caeaf5-3633-4e49-8091-4eae1faddbc2" containerID="da01080f59497260be681c7d244677ea637f6b2632d7d19575a108a1c79a6935" exitCode=0
Jul 28 17:35:32.353020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.352721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5" event={"ID":"72caeaf5-3633-4e49-8091-4eae1faddbc2","Type":"ContainerDied","Data":"da01080f59497260be681c7d244677ea637f6b2632d7d19575a108a1c79a6935"}
Jul 28 17:35:32.357463 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.354613    2560 generic.go:356] "Generic (PLEG): container finished" podID="238652fd-f3a2-4ee9-9b64-533b62078fea" containerID="4b54bd3c643b5a49acf880c5ad7b156538fc8b89f99fa12cfe665db25886b122" exitCode=0
Jul 28 17:35:32.357463 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.354725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2" event={"ID":"238652fd-f3a2-4ee9-9b64-533b62078fea","Type":"ContainerDied","Data":"4b54bd3c643b5a49acf880c5ad7b156538fc8b89f99fa12cfe665db25886b122"}
Jul 28 17:35:32.357463 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.356501    2560 generic.go:356] "Generic (PLEG): container finished" podID="dfec42e3-7c2a-4e36-aa77-6b789c055a93" containerID="7169211bdaeb99fe4f225aea26895d0f6251cc210afda5b9355f6eaa164e0384" exitCode=0
Jul 28 17:35:32.357463 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.356540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk" event={"ID":"dfec42e3-7c2a-4e36-aa77-6b789c055a93","Type":"ContainerDied","Data":"7169211bdaeb99fe4f225aea26895d0f6251cc210afda5b9355f6eaa164e0384"}
Jul 28 17:35:32.441159 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.441136    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:35:32.489731 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.489706    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:35:32.494994 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.494974    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:35:32.513283 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.513256    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8q69\" (UniqueName: \"kubernetes.io/projected/72caeaf5-3633-4e49-8091-4eae1faddbc2-kube-api-access-x8q69\") pod \"72caeaf5-3633-4e49-8091-4eae1faddbc2\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") "
Jul 28 17:35:32.513400 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.513362    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72caeaf5-3633-4e49-8091-4eae1faddbc2-openshift-service-ca\") pod \"72caeaf5-3633-4e49-8091-4eae1faddbc2\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") "
Jul 28 17:35:32.513467 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.513426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls\") pod \"72caeaf5-3633-4e49-8091-4eae1faddbc2\" (UID: \"72caeaf5-3633-4e49-8091-4eae1faddbc2\") "
Jul 28 17:35:32.513701 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.513669    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72caeaf5-3633-4e49-8091-4eae1faddbc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72caeaf5-3633-4e49-8091-4eae1faddbc2" (UID: "72caeaf5-3633-4e49-8091-4eae1faddbc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:32.515480 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.515450    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72caeaf5-3633-4e49-8091-4eae1faddbc2-kube-api-access-x8q69" (OuterVolumeSpecName: "kube-api-access-x8q69") pod "72caeaf5-3633-4e49-8091-4eae1faddbc2" (UID: "72caeaf5-3633-4e49-8091-4eae1faddbc2"). InnerVolumeSpecName "kube-api-access-x8q69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:32.515572 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.515488    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72caeaf5-3633-4e49-8091-4eae1faddbc2" (UID: "72caeaf5-3633-4e49-8091-4eae1faddbc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:35:32.614914 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.614850    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfec42e3-7c2a-4e36-aa77-6b789c055a93-plugins-config-volume\") pod \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\" (UID: \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\") "
Jul 28 17:35:32.614914 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.614887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238652fd-f3a2-4ee9-9b64-533b62078fea-plugins-config-volume\") pod \"238652fd-f3a2-4ee9-9b64-533b62078fea\" (UID: \"238652fd-f3a2-4ee9-9b64-533b62078fea\") "
Jul 28 17:35:32.614914 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.614911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqd9s\" (UniqueName: \"kubernetes.io/projected/238652fd-f3a2-4ee9-9b64-533b62078fea-kube-api-access-rqd9s\") pod \"238652fd-f3a2-4ee9-9b64-533b62078fea\" (UID: \"238652fd-f3a2-4ee9-9b64-533b62078fea\") "
Jul 28 17:35:32.615104 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.614952    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6wzm\" (UniqueName: \"kubernetes.io/projected/dfec42e3-7c2a-4e36-aa77-6b789c055a93-kube-api-access-s6wzm\") pod \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\" (UID: \"dfec42e3-7c2a-4e36-aa77-6b789c055a93\") "
Jul 28 17:35:32.615282 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.615263    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72caeaf5-3633-4e49-8091-4eae1faddbc2-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:35:32.615383 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.615288    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72caeaf5-3633-4e49-8091-4eae1faddbc2-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:35:32.615383 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.615299    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8q69\" (UniqueName: \"kubernetes.io/projected/72caeaf5-3633-4e49-8091-4eae1faddbc2-kube-api-access-x8q69\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:35:32.615383 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.615263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfec42e3-7c2a-4e36-aa77-6b789c055a93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfec42e3-7c2a-4e36-aa77-6b789c055a93" (UID: "dfec42e3-7c2a-4e36-aa77-6b789c055a93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:32.615383 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.615267    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238652fd-f3a2-4ee9-9b64-533b62078fea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "238652fd-f3a2-4ee9-9b64-533b62078fea" (UID: "238652fd-f3a2-4ee9-9b64-533b62078fea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:32.617029 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.617009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238652fd-f3a2-4ee9-9b64-533b62078fea-kube-api-access-rqd9s" (OuterVolumeSpecName: "kube-api-access-rqd9s") pod "238652fd-f3a2-4ee9-9b64-533b62078fea" (UID: "238652fd-f3a2-4ee9-9b64-533b62078fea"). InnerVolumeSpecName "kube-api-access-rqd9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:32.617093 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.617043    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfec42e3-7c2a-4e36-aa77-6b789c055a93-kube-api-access-s6wzm" (OuterVolumeSpecName: "kube-api-access-s6wzm") pod "dfec42e3-7c2a-4e36-aa77-6b789c055a93" (UID: "dfec42e3-7c2a-4e36-aa77-6b789c055a93"). InnerVolumeSpecName "kube-api-access-s6wzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:32.716635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.716593    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfec42e3-7c2a-4e36-aa77-6b789c055a93-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:35:32.716635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.716626    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238652fd-f3a2-4ee9-9b64-533b62078fea-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:35:32.716635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.716639    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqd9s\" (UniqueName: \"kubernetes.io/projected/238652fd-f3a2-4ee9-9b64-533b62078fea-kube-api-access-rqd9s\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:35:32.716862 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:32.716651    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wzm\" (UniqueName: \"kubernetes.io/projected/dfec42e3-7c2a-4e36-aa77-6b789c055a93-kube-api-access-s6wzm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:35:33.362454 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.362425    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"
Jul 28 17:35:33.362881 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.362474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk" event={"ID":"dfec42e3-7c2a-4e36-aa77-6b789c055a93","Type":"ContainerDied","Data":"9de49939929f8a0a31dd26aec29f6e45c382f679b60ee681aa95ae80b2d12350"}
Jul 28 17:35:33.362881 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.362536    2560 scope.go:117] "RemoveContainer" containerID="7169211bdaeb99fe4f225aea26895d0f6251cc210afda5b9355f6eaa164e0384"
Jul 28 17:35:33.363903 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.363751    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"
Jul 28 17:35:33.363903 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.363758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5" event={"ID":"72caeaf5-3633-4e49-8091-4eae1faddbc2","Type":"ContainerDied","Data":"64d40b126e21d4109cf73daae4ecb2d5f475ac9890b2385ca33f4769499eddb0"}
Jul 28 17:35:33.365179 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.365156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2" event={"ID":"238652fd-f3a2-4ee9-9b64-533b62078fea","Type":"ContainerDied","Data":"17ca4e565a82eae8200a347ae969632c6840858ab33708f4aef56c13cac5e0a2"}
Jul 28 17:35:33.365310 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.365181    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"
Jul 28 17:35:33.373522 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.373508    2560 scope.go:117] "RemoveContainer" containerID="da01080f59497260be681c7d244677ea637f6b2632d7d19575a108a1c79a6935"
Jul 28 17:35:33.381244 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.381228    2560 scope.go:117] "RemoveContainer" containerID="4b54bd3c643b5a49acf880c5ad7b156538fc8b89f99fa12cfe665db25886b122"
Jul 28 17:35:33.434555 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.434528    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"]
Jul 28 17:35:33.441048 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.441026    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-949b0a76-5685b8c66b-xpll5"]
Jul 28 17:35:33.467904 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.467872    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"]
Jul 28 17:35:33.475045 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.475016    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-949b0a76-d5f68f684-cv9k2"]
Jul 28 17:35:33.489879 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.489856    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"]
Jul 28 17:35:33.495290 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.495267    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-949b0a76-7559cbdbd-zrbzk"]
Jul 28 17:35:33.872665 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.872625    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="238652fd-f3a2-4ee9-9b64-533b62078fea" path="/var/lib/kubelet/pods/238652fd-f3a2-4ee9-9b64-533b62078fea/volumes"
Jul 28 17:35:33.873090 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.873074    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72caeaf5-3633-4e49-8091-4eae1faddbc2" path="/var/lib/kubelet/pods/72caeaf5-3633-4e49-8091-4eae1faddbc2/volumes"
Jul 28 17:35:33.873470 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:33.873455    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfec42e3-7c2a-4e36-aa77-6b789c055a93" path="/var/lib/kubelet/pods/dfec42e3-7c2a-4e36-aa77-6b789c055a93/volumes"
Jul 28 17:35:44.392381 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392284    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"]
Jul 28 17:35:44.392844 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392828    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72caeaf5-3633-4e49-8091-4eae1faddbc2" containerName="maas-api"
Jul 28 17:35:44.392884 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392846    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="72caeaf5-3633-4e49-8091-4eae1faddbc2" containerName="maas-api"
Jul 28 17:35:44.392884 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392863    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="238652fd-f3a2-4ee9-9b64-533b62078fea" containerName="payload-processing"
Jul 28 17:35:44.392884 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392869    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="238652fd-f3a2-4ee9-9b64-533b62078fea" containerName="payload-processing"
Jul 28 17:35:44.392884 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392882    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfec42e3-7c2a-4e36-aa77-6b789c055a93" containerName="payload-pre-processing"
Jul 28 17:35:44.393001 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392887    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfec42e3-7c2a-4e36-aa77-6b789c055a93" containerName="payload-pre-processing"
Jul 28 17:35:44.393001 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392948    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="238652fd-f3a2-4ee9-9b64-533b62078fea" containerName="payload-processing"
Jul 28 17:35:44.393001 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392958    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfec42e3-7c2a-4e36-aa77-6b789c055a93" containerName="payload-pre-processing"
Jul 28 17:35:44.393001 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.392966    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="72caeaf5-3633-4e49-8091-4eae1faddbc2" containerName="maas-api"
Jul 28 17:35:44.395469 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.395453    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.397838 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.397819    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7053c062-serving-cert\""
Jul 28 17:35:44.397925 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.397885    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:35:44.398519 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.398502    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:35:44.398608 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.398533    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:35:44.408243 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.408223    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"]
Jul 28 17:35:44.533008 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.532973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-maas-api-tls\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.533008 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.533009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-openshift-service-ca\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.533188 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.533091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svcdn\" (UniqueName: \"kubernetes.io/projected/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-kube-api-access-svcdn\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.633552 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.633509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcdn\" (UniqueName: \"kubernetes.io/projected/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-kube-api-access-svcdn\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.633730 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.633619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-maas-api-tls\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.633730 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.633640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-openshift-service-ca\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.634386 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.634332    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-openshift-service-ca\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.635978 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.635959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-maas-api-tls\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.641930 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.641904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svcdn\" (UniqueName: \"kubernetes.io/projected/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-kube-api-access-svcdn\") pod \"maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.706242 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.706172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:44.845048 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:44.845025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"]
Jul 28 17:35:44.847139 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:35:44.847112    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cb2a6c_ae6c_4986_980c_ec4dd8ad025b.slice/crio-87743a92661402ed80c1d8053b124e41df810d2aafe6f387c8d9e5ac6061a0ff WatchSource:0}: Error finding container 87743a92661402ed80c1d8053b124e41df810d2aafe6f387c8d9e5ac6061a0ff: Status 404 returned error can't find the container with id 87743a92661402ed80c1d8053b124e41df810d2aafe6f387c8d9e5ac6061a0ff
Jul 28 17:35:45.351584 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.351551    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"]
Jul 28 17:35:45.353832 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.353809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:45.355422 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.355399    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"]
Jul 28 17:35:45.356214 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.356195    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7053c062\""
Jul 28 17:35:45.357454 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.357439    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:45.375856 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.375834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"]
Jul 28 17:35:45.377606 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.377587    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"]
Jul 28 17:35:45.423049 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.423024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k" event={"ID":"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b","Type":"ContainerStarted","Data":"4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725"}
Jul 28 17:35:45.423049 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.423067    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k" event={"ID":"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b","Type":"ContainerStarted","Data":"87743a92661402ed80c1d8053b124e41df810d2aafe6f387c8d9e5ac6061a0ff"}
Jul 28 17:35:45.423475 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.423095    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:45.446294 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.446243    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k" podStartSLOduration=1.061417924 podStartE2EDuration="1.44622415s" totalImagesPullingTime="384.806226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:45.445637949 +0000 UTC m=+1616.117965870" watchObservedRunningTime="2026-07-28 17:35:45.44622415 +0000 UTC m=+1616.118552075"
Jul 28 17:35:45.542969 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.542906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce414dda-5e7d-4df1-840d-aca3fbd2907f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v\" (UID: \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:45.542969 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.542967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e289ef6d-2b54-4706-9091-32cbc3d9cabd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6\" (UID: \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:45.543159 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.543021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vccgq\" (UniqueName: \"kubernetes.io/projected/ce414dda-5e7d-4df1-840d-aca3fbd2907f-kube-api-access-vccgq\") pod \"payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v\" (UID: \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:45.543159 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.543042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njm6k\" (UniqueName: \"kubernetes.io/projected/e289ef6d-2b54-4706-9091-32cbc3d9cabd-kube-api-access-njm6k\") pod \"payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6\" (UID: \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:45.643585 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.643554    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce414dda-5e7d-4df1-840d-aca3fbd2907f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v\" (UID: \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:45.643702 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.643628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e289ef6d-2b54-4706-9091-32cbc3d9cabd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6\" (UID: \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:45.643702 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.643678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vccgq\" (UniqueName: \"kubernetes.io/projected/ce414dda-5e7d-4df1-840d-aca3fbd2907f-kube-api-access-vccgq\") pod \"payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v\" (UID: \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:45.643821 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.643707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njm6k\" (UniqueName: \"kubernetes.io/projected/e289ef6d-2b54-4706-9091-32cbc3d9cabd-kube-api-access-njm6k\") pod \"payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6\" (UID: \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:45.644280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.644263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e289ef6d-2b54-4706-9091-32cbc3d9cabd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6\" (UID: \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:45.644280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.644276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce414dda-5e7d-4df1-840d-aca3fbd2907f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v\" (UID: \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:45.656590 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.656568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vccgq\" (UniqueName: \"kubernetes.io/projected/ce414dda-5e7d-4df1-840d-aca3fbd2907f-kube-api-access-vccgq\") pod \"payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v\" (UID: \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:45.656705 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.656608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njm6k\" (UniqueName: \"kubernetes.io/projected/e289ef6d-2b54-4706-9091-32cbc3d9cabd-kube-api-access-njm6k\") pod \"payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6\" (UID: \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:45.665448 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.665426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:45.670394 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.670377    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:45.810157 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.810133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"]
Jul 28 17:35:45.812530 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:35:45.812506    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode289ef6d_2b54_4706_9091_32cbc3d9cabd.slice/crio-de939abfbb744b1c3e4a55613c0178f5a8de2895c36384b3d5e713eec166f537 WatchSource:0}: Error finding container de939abfbb744b1c3e4a55613c0178f5a8de2895c36384b3d5e713eec166f537: Status 404 returned error can't find the container with id de939abfbb744b1c3e4a55613c0178f5a8de2895c36384b3d5e713eec166f537
Jul 28 17:35:45.831951 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:45.831926    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"]
Jul 28 17:35:45.834172 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:35:45.834080    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce414dda_5e7d_4df1_840d_aca3fbd2907f.slice/crio-dccd396059498e742ff68c4aa9e63e3be5af618845671e2949826350a0a7ba21 WatchSource:0}: Error finding container dccd396059498e742ff68c4aa9e63e3be5af618845671e2949826350a0a7ba21: Status 404 returned error can't find the container with id dccd396059498e742ff68c4aa9e63e3be5af618845671e2949826350a0a7ba21
Jul 28 17:35:46.430446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:46.430409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6" event={"ID":"e289ef6d-2b54-4706-9091-32cbc3d9cabd","Type":"ContainerStarted","Data":"bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294"}
Jul 28 17:35:46.430913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:46.430472    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:46.430913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:46.430494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6" event={"ID":"e289ef6d-2b54-4706-9091-32cbc3d9cabd","Type":"ContainerStarted","Data":"de939abfbb744b1c3e4a55613c0178f5a8de2895c36384b3d5e713eec166f537"}
Jul 28 17:35:46.431980 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:46.431954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v" event={"ID":"ce414dda-5e7d-4df1-840d-aca3fbd2907f","Type":"ContainerStarted","Data":"343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690"}
Jul 28 17:35:46.432096 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:46.432002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v" event={"ID":"ce414dda-5e7d-4df1-840d-aca3fbd2907f","Type":"ContainerStarted","Data":"dccd396059498e742ff68c4aa9e63e3be5af618845671e2949826350a0a7ba21"}
Jul 28 17:35:46.432149 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:46.432133    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:35:46.452564 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:46.452517    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6" podStartSLOduration=1.452502346 podStartE2EDuration="1.452502346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:46.450502739 +0000 UTC m=+1617.122830661" watchObservedRunningTime="2026-07-28 17:35:46.452502346 +0000 UTC m=+1617.124830265"
Jul 28 17:35:46.469386 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:46.469319    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v" podStartSLOduration=1.469305106 podStartE2EDuration="1.469305106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:46.468952844 +0000 UTC m=+1617.141280765" watchObservedRunningTime="2026-07-28 17:35:46.469305106 +0000 UTC m=+1617.141633028"
Jul 28 17:35:51.437458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:51.437426    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:35:57.438504 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:57.438475    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:35:57.438894 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:35:57.438592    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:36:19.968181 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:19.968136    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"]
Jul 28 17:36:19.968715 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:19.968394    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k" podUID="77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" containerName="maas-api" containerID="cri-o://4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725" gracePeriod=30
Jul 28 17:36:20.110262 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.110224    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"]
Jul 28 17:36:20.110554 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.110502    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6" podUID="e289ef6d-2b54-4706-9091-32cbc3d9cabd" containerName="payload-processing" containerID="cri-o://bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294" gracePeriod=30
Jul 28 17:36:20.119424 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.117830    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"]
Jul 28 17:36:20.119424 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.118032    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v" podUID="ce414dda-5e7d-4df1-840d-aca3fbd2907f" containerName="payload-pre-processing" containerID="cri-o://343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690" gracePeriod=30
Jul 28 17:36:20.245113 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.245084    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:36:20.356387 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.356359    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-openshift-service-ca\") pod \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") "
Jul 28 17:36:20.356542 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.356402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-maas-api-tls\") pod \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") "
Jul 28 17:36:20.356542 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.356519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svcdn\" (UniqueName: \"kubernetes.io/projected/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-kube-api-access-svcdn\") pod \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\" (UID: \"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b\") "
Jul 28 17:36:20.356824 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.356798    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" (UID: "77cb2a6c-ae6c-4986-980c-ec4dd8ad025b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:20.360584 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.360558    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" (UID: "77cb2a6c-ae6c-4986-980c-ec4dd8ad025b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:20.361790 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.361769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-kube-api-access-svcdn" (OuterVolumeSpecName: "kube-api-access-svcdn") pod "77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" (UID: "77cb2a6c-ae6c-4986-980c-ec4dd8ad025b"). InnerVolumeSpecName "kube-api-access-svcdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:20.377121 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.377093    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:36:20.383479 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.383456    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:36:20.457063 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.457035    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njm6k\" (UniqueName: \"kubernetes.io/projected/e289ef6d-2b54-4706-9091-32cbc3d9cabd-kube-api-access-njm6k\") pod \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\" (UID: \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\") "
Jul 28 17:36:20.457216 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.457155    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e289ef6d-2b54-4706-9091-32cbc3d9cabd-plugins-config-volume\") pod \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\" (UID: \"e289ef6d-2b54-4706-9091-32cbc3d9cabd\") "
Jul 28 17:36:20.457469 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.457452    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svcdn\" (UniqueName: \"kubernetes.io/projected/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-kube-api-access-svcdn\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:36:20.457521 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.457472    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:36:20.457521 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.457482    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:36:20.457521 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.457485    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e289ef6d-2b54-4706-9091-32cbc3d9cabd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e289ef6d-2b54-4706-9091-32cbc3d9cabd" (UID: "e289ef6d-2b54-4706-9091-32cbc3d9cabd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:20.459052 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.459035    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e289ef6d-2b54-4706-9091-32cbc3d9cabd-kube-api-access-njm6k" (OuterVolumeSpecName: "kube-api-access-njm6k") pod "e289ef6d-2b54-4706-9091-32cbc3d9cabd" (UID: "e289ef6d-2b54-4706-9091-32cbc3d9cabd"). InnerVolumeSpecName "kube-api-access-njm6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:20.558539 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.558519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce414dda-5e7d-4df1-840d-aca3fbd2907f-plugins-config-volume\") pod \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\" (UID: \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\") "
Jul 28 17:36:20.558655 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.558575    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vccgq\" (UniqueName: \"kubernetes.io/projected/ce414dda-5e7d-4df1-840d-aca3fbd2907f-kube-api-access-vccgq\") pod \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\" (UID: \"ce414dda-5e7d-4df1-840d-aca3fbd2907f\") "
Jul 28 17:36:20.558726 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.558715    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e289ef6d-2b54-4706-9091-32cbc3d9cabd-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:36:20.558760 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.558728    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njm6k\" (UniqueName: \"kubernetes.io/projected/e289ef6d-2b54-4706-9091-32cbc3d9cabd-kube-api-access-njm6k\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:36:20.558853 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.558834    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce414dda-5e7d-4df1-840d-aca3fbd2907f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce414dda-5e7d-4df1-840d-aca3fbd2907f" (UID: "ce414dda-5e7d-4df1-840d-aca3fbd2907f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:20.560465 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.560447    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce414dda-5e7d-4df1-840d-aca3fbd2907f-kube-api-access-vccgq" (OuterVolumeSpecName: "kube-api-access-vccgq") pod "ce414dda-5e7d-4df1-840d-aca3fbd2907f" (UID: "ce414dda-5e7d-4df1-840d-aca3fbd2907f"). InnerVolumeSpecName "kube-api-access-vccgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:20.592957 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.592926    2560 generic.go:356] "Generic (PLEG): container finished" podID="77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" containerID="4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725" exitCode=0
Jul 28 17:36:20.593054 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.592972    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"
Jul 28 17:36:20.593054 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.593012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k" event={"ID":"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b","Type":"ContainerDied","Data":"4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725"}
Jul 28 17:36:20.593149 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.593057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k" event={"ID":"77cb2a6c-ae6c-4986-980c-ec4dd8ad025b","Type":"ContainerDied","Data":"87743a92661402ed80c1d8053b124e41df810d2aafe6f387c8d9e5ac6061a0ff"}
Jul 28 17:36:20.593149 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.593084    2560 scope.go:117] "RemoveContainer" containerID="4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725"
Jul 28 17:36:20.594332 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.594304    2560 generic.go:356] "Generic (PLEG): container finished" podID="e289ef6d-2b54-4706-9091-32cbc3d9cabd" containerID="bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294" exitCode=0
Jul 28 17:36:20.594458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.594374    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"
Jul 28 17:36:20.594458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.594399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6" event={"ID":"e289ef6d-2b54-4706-9091-32cbc3d9cabd","Type":"ContainerDied","Data":"bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294"}
Jul 28 17:36:20.594458 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.594458    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6" event={"ID":"e289ef6d-2b54-4706-9091-32cbc3d9cabd","Type":"ContainerDied","Data":"de939abfbb744b1c3e4a55613c0178f5a8de2895c36384b3d5e713eec166f537"}
Jul 28 17:36:20.595904 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.595870    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce414dda-5e7d-4df1-840d-aca3fbd2907f" containerID="343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690" exitCode=0
Jul 28 17:36:20.595904 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.595893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v" event={"ID":"ce414dda-5e7d-4df1-840d-aca3fbd2907f","Type":"ContainerDied","Data":"343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690"}
Jul 28 17:36:20.596063 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.595926    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v" event={"ID":"ce414dda-5e7d-4df1-840d-aca3fbd2907f","Type":"ContainerDied","Data":"dccd396059498e742ff68c4aa9e63e3be5af618845671e2949826350a0a7ba21"}
Jul 28 17:36:20.596063 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.595932    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"
Jul 28 17:36:20.609299 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.609273    2560 scope.go:117] "RemoveContainer" containerID="4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725"
Jul 28 17:36:20.609598 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:36:20.609576    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725\": container with ID starting with 4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725 not found: ID does not exist" containerID="4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725"
Jul 28 17:36:20.609692 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.609603    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725"} err="failed to get container status \"4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725\": rpc error: code = NotFound desc = could not find container \"4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725\": container with ID starting with 4e0ee88830359bea4a2585c44739ff730e2d54d43631db415b4628cf8c2be725 not found: ID does not exist"
Jul 28 17:36:20.609692 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.609619    2560 scope.go:117] "RemoveContainer" containerID="bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294"
Jul 28 17:36:20.619617 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.619599    2560 scope.go:117] "RemoveContainer" containerID="bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294"
Jul 28 17:36:20.619989 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:36:20.619969    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294\": container with ID starting with bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294 not found: ID does not exist" containerID="bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294"
Jul 28 17:36:20.620065 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.619992    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294"} err="failed to get container status \"bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294\": rpc error: code = NotFound desc = could not find container \"bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294\": container with ID starting with bfa842f3311774b269375dad8f11accef1d0de8eb2e26009b69de75eed8de294 not found: ID does not exist"
Jul 28 17:36:20.620065 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.620008    2560 scope.go:117] "RemoveContainer" containerID="343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690"
Jul 28 17:36:20.625280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.625255    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"]
Jul 28 17:36:20.628365 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.628334    2560 scope.go:117] "RemoveContainer" containerID="343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690"
Jul 28 17:36:20.628624 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:36:20.628603    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690\": container with ID starting with 343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690 not found: ID does not exist" containerID="343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690"
Jul 28 17:36:20.628694 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.628634    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690"} err="failed to get container status \"343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690\": rpc error: code = NotFound desc = could not find container \"343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690\": container with ID starting with 343208f73eb89854ee751a0d1a657549193641b79f949b6222ff0c3c24057690 not found: ID does not exist"
Jul 28 17:36:20.629368 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.629325    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7053c062-84f45b756f-r2d8k"]
Jul 28 17:36:20.641179 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.641157    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"]
Jul 28 17:36:20.645469 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.645446    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7053c062-6dc9bdd9c7-v94s6"]
Jul 28 17:36:20.657487 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.657469    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"]
Jul 28 17:36:20.659796 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.659779    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vccgq\" (UniqueName: \"kubernetes.io/projected/ce414dda-5e7d-4df1-840d-aca3fbd2907f-kube-api-access-vccgq\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:36:20.659887 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.659797    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce414dda-5e7d-4df1-840d-aca3fbd2907f-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:36:20.660036 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:20.660021    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7053c062-697f946d6d-7m84v"]
Jul 28 17:36:21.872315 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:21.872282    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" path="/var/lib/kubelet/pods/77cb2a6c-ae6c-4986-980c-ec4dd8ad025b/volumes"
Jul 28 17:36:21.872682 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:21.872622    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce414dda-5e7d-4df1-840d-aca3fbd2907f" path="/var/lib/kubelet/pods/ce414dda-5e7d-4df1-840d-aca3fbd2907f/volumes"
Jul 28 17:36:21.872913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:21.872901    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e289ef6d-2b54-4706-9091-32cbc3d9cabd" path="/var/lib/kubelet/pods/e289ef6d-2b54-4706-9091-32cbc3d9cabd/volumes"
Jul 28 17:36:27.992439 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.992408    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"]
Jul 28 17:36:27.992971 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.992952    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" containerName="maas-api"
Jul 28 17:36:27.993093 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.992974    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" containerName="maas-api"
Jul 28 17:36:27.993093 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.992999    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e289ef6d-2b54-4706-9091-32cbc3d9cabd" containerName="payload-processing"
Jul 28 17:36:27.993093 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.993009    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e289ef6d-2b54-4706-9091-32cbc3d9cabd" containerName="payload-processing"
Jul 28 17:36:27.993093 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.993037    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce414dda-5e7d-4df1-840d-aca3fbd2907f" containerName="payload-pre-processing"
Jul 28 17:36:27.993093 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.993046    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce414dda-5e7d-4df1-840d-aca3fbd2907f" containerName="payload-pre-processing"
Jul 28 17:36:27.993241 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.993136    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="77cb2a6c-ae6c-4986-980c-ec4dd8ad025b" containerName="maas-api"
Jul 28 17:36:27.993241 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.993150    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e289ef6d-2b54-4706-9091-32cbc3d9cabd" containerName="payload-processing"
Jul 28 17:36:27.993241 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.993160    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce414dda-5e7d-4df1-840d-aca3fbd2907f" containerName="payload-pre-processing"
Jul 28 17:36:27.996152 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:27.996132    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.006780 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.006760    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:36:28.007064 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.007049    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:36:28.007572 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.007551    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:36:28.007612 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.007572    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-133ebd2a-serving-cert\""
Jul 28 17:36:28.013554 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.013535    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"]
Jul 28 17:36:28.125582 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.125551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-maas-api-tls\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.125582 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.125580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-openshift-service-ca\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.125758 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.125661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsl9m\" (UniqueName: \"kubernetes.io/projected/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-kube-api-access-xsl9m\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.226262 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.226234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-maas-api-tls\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.226447 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.226269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-openshift-service-ca\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.226447 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.226380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsl9m\" (UniqueName: \"kubernetes.io/projected/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-kube-api-access-xsl9m\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.226961 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.226940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-openshift-service-ca\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.228655 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.228636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-maas-api-tls\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.245540 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.245480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsl9m\" (UniqueName: \"kubernetes.io/projected/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-kube-api-access-xsl9m\") pod \"maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.305385 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.305333    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:28.432952 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.432917    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"]
Jul 28 17:36:28.435498 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:36:28.435469    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfab1ce_2fbc_4d8b_8daa_09d99a08ee0e.slice/crio-f3700fd5d540bffa6901e3c4966bdcd5c2d14266bdad804447f3f39cf9335ec1 WatchSource:0}: Error finding container f3700fd5d540bffa6901e3c4966bdcd5c2d14266bdad804447f3f39cf9335ec1: Status 404 returned error can't find the container with id f3700fd5d540bffa6901e3c4966bdcd5c2d14266bdad804447f3f39cf9335ec1
Jul 28 17:36:28.638398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.638360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7" event={"ID":"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e","Type":"ContainerStarted","Data":"f3700fd5d540bffa6901e3c4966bdcd5c2d14266bdad804447f3f39cf9335ec1"}
Jul 28 17:36:28.782746 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.782712    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"]
Jul 28 17:36:28.785398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.785383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:28.787779 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.787761    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-133ebd2a\""
Jul 28 17:36:28.787885 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.787841    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-133ebd2a-dockercfg-zgprv\""
Jul 28 17:36:28.799332 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.799308    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"]
Jul 28 17:36:28.828783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.828759    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"]
Jul 28 17:36:28.830870 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.830853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:28.831310 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.831286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75v6f\" (UniqueName: \"kubernetes.io/projected/948f2520-5da7-4dfd-946a-340632df0a93-kube-api-access-75v6f\") pod \"payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs\" (UID: \"948f2520-5da7-4dfd-946a-340632df0a93\") " pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:28.831453 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.831336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5vd\" (UniqueName: \"kubernetes.io/projected/e9fe1695-60c2-4009-97aa-57daf069e3e6-kube-api-access-dt5vd\") pod \"payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm\" (UID: \"e9fe1695-60c2-4009-97aa-57daf069e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:28.831453 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.831420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948f2520-5da7-4dfd-946a-340632df0a93-plugins-config-volume\") pod \"payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs\" (UID: \"948f2520-5da7-4dfd-946a-340632df0a93\") " pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:28.831570 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.831460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fe1695-60c2-4009-97aa-57daf069e3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm\" (UID: \"e9fe1695-60c2-4009-97aa-57daf069e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:28.842846 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.842827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"]
Jul 28 17:36:28.932291 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.932211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948f2520-5da7-4dfd-946a-340632df0a93-plugins-config-volume\") pod \"payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs\" (UID: \"948f2520-5da7-4dfd-946a-340632df0a93\") " pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:28.932291 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.932262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fe1695-60c2-4009-97aa-57daf069e3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm\" (UID: \"e9fe1695-60c2-4009-97aa-57daf069e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:28.932533 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.932314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75v6f\" (UniqueName: \"kubernetes.io/projected/948f2520-5da7-4dfd-946a-340632df0a93-kube-api-access-75v6f\") pod \"payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs\" (UID: \"948f2520-5da7-4dfd-946a-340632df0a93\") " pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:28.932533 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.932404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5vd\" (UniqueName: \"kubernetes.io/projected/e9fe1695-60c2-4009-97aa-57daf069e3e6-kube-api-access-dt5vd\") pod \"payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm\" (UID: \"e9fe1695-60c2-4009-97aa-57daf069e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:28.933112 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.933089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fe1695-60c2-4009-97aa-57daf069e3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm\" (UID: \"e9fe1695-60c2-4009-97aa-57daf069e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:28.933148 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.933089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948f2520-5da7-4dfd-946a-340632df0a93-plugins-config-volume\") pod \"payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs\" (UID: \"948f2520-5da7-4dfd-946a-340632df0a93\") " pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:28.941526 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.941499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5vd\" (UniqueName: \"kubernetes.io/projected/e9fe1695-60c2-4009-97aa-57daf069e3e6-kube-api-access-dt5vd\") pod \"payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm\" (UID: \"e9fe1695-60c2-4009-97aa-57daf069e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:28.941632 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:28.941543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75v6f\" (UniqueName: \"kubernetes.io/projected/948f2520-5da7-4dfd-946a-340632df0a93-kube-api-access-75v6f\") pod \"payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs\" (UID: \"948f2520-5da7-4dfd-946a-340632df0a93\") " pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:29.094994 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.094964    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:29.142474 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.142405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:29.240485 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.240454    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"]
Jul 28 17:36:29.241984 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:36:29.241959    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948f2520_5da7_4dfd_946a_340632df0a93.slice/crio-bdcc38b1d53e633b8c6c4b7cb1b12a99742e62cc5a785dd8a9926b7817eec649 WatchSource:0}: Error finding container bdcc38b1d53e633b8c6c4b7cb1b12a99742e62cc5a785dd8a9926b7817eec649: Status 404 returned error can't find the container with id bdcc38b1d53e633b8c6c4b7cb1b12a99742e62cc5a785dd8a9926b7817eec649
Jul 28 17:36:29.288146 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.288116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"]
Jul 28 17:36:29.291579 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:36:29.291555    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fe1695_60c2_4009_97aa_57daf069e3e6.slice/crio-8f7308b7923f806dc48463cdff851612606dc33569c4e14f0c2c335866a048bb WatchSource:0}: Error finding container 8f7308b7923f806dc48463cdff851612606dc33569c4e14f0c2c335866a048bb: Status 404 returned error can't find the container with id 8f7308b7923f806dc48463cdff851612606dc33569c4e14f0c2c335866a048bb
Jul 28 17:36:29.650871 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.650835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs" event={"ID":"948f2520-5da7-4dfd-946a-340632df0a93","Type":"ContainerStarted","Data":"32ee3a8d5960b63bc1bf45a9225a2a8bb29842feaa214af3827ef3c7af4c83aa"}
Jul 28 17:36:29.651059 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.650926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:29.651059 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.650955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs" event={"ID":"948f2520-5da7-4dfd-946a-340632df0a93","Type":"ContainerStarted","Data":"bdcc38b1d53e633b8c6c4b7cb1b12a99742e62cc5a785dd8a9926b7817eec649"}
Jul 28 17:36:29.652709 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.652675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7" event={"ID":"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e","Type":"ContainerStarted","Data":"b585396a18bdc3a0c91b8a5f6557e42f6391a86c04a13bd03af6f621ac1c8b1c"}
Jul 28 17:36:29.652831 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.652765    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:29.654175 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.654153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm" event={"ID":"e9fe1695-60c2-4009-97aa-57daf069e3e6","Type":"ContainerStarted","Data":"e8183f10a91f66c82e290d12175dc5cb33dd1e5e5a7dba6d93935a7b829795f8"}
Jul 28 17:36:29.654285 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.654188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm" event={"ID":"e9fe1695-60c2-4009-97aa-57daf069e3e6","Type":"ContainerStarted","Data":"8f7308b7923f806dc48463cdff851612606dc33569c4e14f0c2c335866a048bb"}
Jul 28 17:36:29.654285 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.654225    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:36:29.677603 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.677558    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs" podStartSLOduration=1.6775441180000001 podStartE2EDuration="1.677544118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:29.67541336 +0000 UTC m=+1660.347741277" watchObservedRunningTime="2026-07-28 17:36:29.677544118 +0000 UTC m=+1660.349872017"
Jul 28 17:36:29.693250 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.693204    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7" podStartSLOduration=2.2938107580000002 podStartE2EDuration="2.693190591s" totalImagesPullingTime="399.379833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:29.69220091 +0000 UTC m=+1660.364528843" watchObservedRunningTime="2026-07-28 17:36:29.693190591 +0000 UTC m=+1660.365518510"
Jul 28 17:36:29.711504 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:29.711465    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm" podStartSLOduration=1.711452869 podStartE2EDuration="1.711452869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:29.710759471 +0000 UTC m=+1660.383087404" watchObservedRunningTime="2026-07-28 17:36:29.711452869 +0000 UTC m=+1660.383780788"
Jul 28 17:36:35.664473 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:35.664441    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:36:40.661428 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:40.661399    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:36:40.661826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:36:40.661468    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:37:04.526413 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.526383    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"]
Jul 28 17:37:04.526873 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.526651    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7" podUID="abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" containerName="maas-api" containerID="cri-o://b585396a18bdc3a0c91b8a5f6557e42f6391a86c04a13bd03af6f621ac1c8b1c" gracePeriod=30
Jul 28 17:37:04.586550 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.586523    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"]
Jul 28 17:37:04.586777 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.586736    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs" podUID="948f2520-5da7-4dfd-946a-340632df0a93" containerName="payload-processing" containerID="cri-o://32ee3a8d5960b63bc1bf45a9225a2a8bb29842feaa214af3827ef3c7af4c83aa" gracePeriod=30
Jul 28 17:37:04.606265 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.606242    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"]
Jul 28 17:37:04.606455 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.606436    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm" podUID="e9fe1695-60c2-4009-97aa-57daf069e3e6" containerName="payload-pre-processing" containerID="cri-o://e8183f10a91f66c82e290d12175dc5cb33dd1e5e5a7dba6d93935a7b829795f8" gracePeriod=30
Jul 28 17:37:04.823945 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.823910    2560 generic.go:356] "Generic (PLEG): container finished" podID="e9fe1695-60c2-4009-97aa-57daf069e3e6" containerID="e8183f10a91f66c82e290d12175dc5cb33dd1e5e5a7dba6d93935a7b829795f8" exitCode=0
Jul 28 17:37:04.823945 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.823934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm" event={"ID":"e9fe1695-60c2-4009-97aa-57daf069e3e6","Type":"ContainerDied","Data":"e8183f10a91f66c82e290d12175dc5cb33dd1e5e5a7dba6d93935a7b829795f8"}
Jul 28 17:37:04.825904 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.825867    2560 generic.go:356] "Generic (PLEG): container finished" podID="948f2520-5da7-4dfd-946a-340632df0a93" containerID="32ee3a8d5960b63bc1bf45a9225a2a8bb29842feaa214af3827ef3c7af4c83aa" exitCode=0
Jul 28 17:37:04.826021 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.825959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs" event={"ID":"948f2520-5da7-4dfd-946a-340632df0a93","Type":"ContainerDied","Data":"32ee3a8d5960b63bc1bf45a9225a2a8bb29842feaa214af3827ef3c7af4c83aa"}
Jul 28 17:37:04.827842 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.827818    2560 generic.go:356] "Generic (PLEG): container finished" podID="abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" containerID="b585396a18bdc3a0c91b8a5f6557e42f6391a86c04a13bd03af6f621ac1c8b1c" exitCode=0
Jul 28 17:37:04.827967 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.827852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7" event={"ID":"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e","Type":"ContainerDied","Data":"b585396a18bdc3a0c91b8a5f6557e42f6391a86c04a13bd03af6f621ac1c8b1c"}
Jul 28 17:37:04.858653 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.858634    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:37:04.869782 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.869759    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:37:04.875597 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.875577    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:37:04.958455 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.958425    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75v6f\" (UniqueName: \"kubernetes.io/projected/948f2520-5da7-4dfd-946a-340632df0a93-kube-api-access-75v6f\") pod \"948f2520-5da7-4dfd-946a-340632df0a93\" (UID: \"948f2520-5da7-4dfd-946a-340632df0a93\") "
Jul 28 17:37:04.958608 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.958469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948f2520-5da7-4dfd-946a-340632df0a93-plugins-config-volume\") pod \"948f2520-5da7-4dfd-946a-340632df0a93\" (UID: \"948f2520-5da7-4dfd-946a-340632df0a93\") "
Jul 28 17:37:04.958835 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.958813    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948f2520-5da7-4dfd-946a-340632df0a93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "948f2520-5da7-4dfd-946a-340632df0a93" (UID: "948f2520-5da7-4dfd-946a-340632df0a93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:04.960430 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:04.960412    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948f2520-5da7-4dfd-946a-340632df0a93-kube-api-access-75v6f" (OuterVolumeSpecName: "kube-api-access-75v6f") pod "948f2520-5da7-4dfd-946a-340632df0a93" (UID: "948f2520-5da7-4dfd-946a-340632df0a93"). InnerVolumeSpecName "kube-api-access-75v6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:05.059826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.059800    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-openshift-service-ca\") pod \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") "
Jul 28 17:37:05.059826 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.059829    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsl9m\" (UniqueName: \"kubernetes.io/projected/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-kube-api-access-xsl9m\") pod \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") "
Jul 28 17:37:05.060025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.059872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt5vd\" (UniqueName: \"kubernetes.io/projected/e9fe1695-60c2-4009-97aa-57daf069e3e6-kube-api-access-dt5vd\") pod \"e9fe1695-60c2-4009-97aa-57daf069e3e6\" (UID: \"e9fe1695-60c2-4009-97aa-57daf069e3e6\") "
Jul 28 17:37:05.060025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.059915    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-maas-api-tls\") pod \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\" (UID: \"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e\") "
Jul 28 17:37:05.060025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.059940    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fe1695-60c2-4009-97aa-57daf069e3e6-plugins-config-volume\") pod \"e9fe1695-60c2-4009-97aa-57daf069e3e6\" (UID: \"e9fe1695-60c2-4009-97aa-57daf069e3e6\") "
Jul 28 17:37:05.060189 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.060157    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" (UID: "abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:05.060468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.060434    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fe1695-60c2-4009-97aa-57daf069e3e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9fe1695-60c2-4009-97aa-57daf069e3e6" (UID: "e9fe1695-60c2-4009-97aa-57daf069e3e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:05.060468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.060444    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:05.060630 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.060475    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75v6f\" (UniqueName: \"kubernetes.io/projected/948f2520-5da7-4dfd-946a-340632df0a93-kube-api-access-75v6f\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:05.060630 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.060486    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/948f2520-5da7-4dfd-946a-340632df0a93-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:05.061817 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.061801    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-kube-api-access-xsl9m" (OuterVolumeSpecName: "kube-api-access-xsl9m") pod "abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" (UID: "abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e"). InnerVolumeSpecName "kube-api-access-xsl9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:05.062306 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.062280    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" (UID: "abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:05.062485 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.062311    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fe1695-60c2-4009-97aa-57daf069e3e6-kube-api-access-dt5vd" (OuterVolumeSpecName: "kube-api-access-dt5vd") pod "e9fe1695-60c2-4009-97aa-57daf069e3e6" (UID: "e9fe1695-60c2-4009-97aa-57daf069e3e6"). InnerVolumeSpecName "kube-api-access-dt5vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:05.161541 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.161512    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsl9m\" (UniqueName: \"kubernetes.io/projected/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-kube-api-access-xsl9m\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:05.161541 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.161534    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5vd\" (UniqueName: \"kubernetes.io/projected/e9fe1695-60c2-4009-97aa-57daf069e3e6-kube-api-access-dt5vd\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:05.161541 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.161544    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:05.161713 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.161554    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fe1695-60c2-4009-97aa-57daf069e3e6-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:05.833985 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.833950    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"
Jul 28 17:37:05.834439 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.833980    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs" event={"ID":"948f2520-5da7-4dfd-946a-340632df0a93","Type":"ContainerDied","Data":"bdcc38b1d53e633b8c6c4b7cb1b12a99742e62cc5a785dd8a9926b7817eec649"}
Jul 28 17:37:05.834439 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.834041    2560 scope.go:117] "RemoveContainer" containerID="32ee3a8d5960b63bc1bf45a9225a2a8bb29842feaa214af3827ef3c7af4c83aa"
Jul 28 17:37:05.835296 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.835281    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"
Jul 28 17:37:05.835375 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.835319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7" event={"ID":"abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e","Type":"ContainerDied","Data":"f3700fd5d540bffa6901e3c4966bdcd5c2d14266bdad804447f3f39cf9335ec1"}
Jul 28 17:37:05.836992 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.836961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm" event={"ID":"e9fe1695-60c2-4009-97aa-57daf069e3e6","Type":"ContainerDied","Data":"8f7308b7923f806dc48463cdff851612606dc33569c4e14f0c2c335866a048bb"}
Jul 28 17:37:05.836992 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.836977    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"
Jul 28 17:37:05.843790 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.843768    2560 scope.go:117] "RemoveContainer" containerID="b585396a18bdc3a0c91b8a5f6557e42f6391a86c04a13bd03af6f621ac1c8b1c"
Jul 28 17:37:05.852375 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.852355    2560 scope.go:117] "RemoveContainer" containerID="e8183f10a91f66c82e290d12175dc5cb33dd1e5e5a7dba6d93935a7b829795f8"
Jul 28 17:37:05.861030 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.861012    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"]
Jul 28 17:37:05.866431 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.866406    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-133ebd2a-6647c4bb66-lh4cs"]
Jul 28 17:37:05.871859 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.871839    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="948f2520-5da7-4dfd-946a-340632df0a93" path="/var/lib/kubelet/pods/948f2520-5da7-4dfd-946a-340632df0a93/volumes"
Jul 28 17:37:05.878064 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.878048    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"]
Jul 28 17:37:05.883265 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.883246    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-133ebd2a-8667d89bd4-46jkm"]
Jul 28 17:37:05.895297 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.895279    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"]
Jul 28 17:37:05.900463 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:05.900444    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-133ebd2a-5789d76c89-kdmx7"]
Jul 28 17:37:07.872181 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:07.872146    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" path="/var/lib/kubelet/pods/abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e/volumes"
Jul 28 17:37:07.872622 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:07.872492    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9fe1695-60c2-4009-97aa-57daf069e3e6" path="/var/lib/kubelet/pods/e9fe1695-60c2-4009-97aa-57daf069e3e6/volumes"
Jul 28 17:37:13.644898 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.644825    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"]
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645216    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" containerName="maas-api"
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645228    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" containerName="maas-api"
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645240    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9fe1695-60c2-4009-97aa-57daf069e3e6" containerName="payload-pre-processing"
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645246    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9fe1695-60c2-4009-97aa-57daf069e3e6" containerName="payload-pre-processing"
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645257    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="948f2520-5da7-4dfd-946a-340632df0a93" containerName="payload-processing"
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645262    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="948f2520-5da7-4dfd-946a-340632df0a93" containerName="payload-processing"
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645372    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9fe1695-60c2-4009-97aa-57daf069e3e6" containerName="payload-pre-processing"
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645383    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="948f2520-5da7-4dfd-946a-340632df0a93" containerName="payload-processing"
Jul 28 17:37:13.647171 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.645395    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfab1ce-2fbc-4d8b-8daa-09d99a08ee0e" containerName="maas-api"
Jul 28 17:37:13.648279 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.648261    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.651333 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.651316    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:37:13.651462 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.651432    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:37:13.652235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.652213    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-31d2617b-serving-cert\""
Jul 28 17:37:13.652315 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.652243    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:37:13.658693 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.658674    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"]
Jul 28 17:37:13.729602 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.729568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51c837e-f224-4f7d-8f32-ac91de903074-maas-api-tls\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.729731 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.729609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjnf\" (UniqueName: \"kubernetes.io/projected/a51c837e-f224-4f7d-8f32-ac91de903074-kube-api-access-ppjnf\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.729731 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.729681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51c837e-f224-4f7d-8f32-ac91de903074-openshift-service-ca\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.830544 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.830514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51c837e-f224-4f7d-8f32-ac91de903074-openshift-service-ca\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.830696 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.830578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51c837e-f224-4f7d-8f32-ac91de903074-maas-api-tls\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.830696 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.830622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjnf\" (UniqueName: \"kubernetes.io/projected/a51c837e-f224-4f7d-8f32-ac91de903074-kube-api-access-ppjnf\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.831207 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.831190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51c837e-f224-4f7d-8f32-ac91de903074-openshift-service-ca\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.832897 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.832871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51c837e-f224-4f7d-8f32-ac91de903074-maas-api-tls\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.844690 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.844667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjnf\" (UniqueName: \"kubernetes.io/projected/a51c837e-f224-4f7d-8f32-ac91de903074-kube-api-access-ppjnf\") pod \"maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:13.958488 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:13.958417    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:14.088980 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.088955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"]
Jul 28 17:37:14.091665 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:37:14.091636    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51c837e_f224_4f7d_8f32_ac91de903074.slice/crio-dec38b22343682cb93b860f449928bdce937afc39516014fe3412e4da8f2122a WatchSource:0}: Error finding container dec38b22343682cb93b860f449928bdce937afc39516014fe3412e4da8f2122a: Status 404 returned error can't find the container with id dec38b22343682cb93b860f449928bdce937afc39516014fe3412e4da8f2122a
Jul 28 17:37:14.484005 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.483973    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"]
Jul 28 17:37:14.488461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.488440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:14.492478 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.492460    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-31d2617b\""
Jul 28 17:37:14.505604 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.505581    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"]
Jul 28 17:37:14.517835 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.517812    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"]
Jul 28 17:37:14.521268 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.521250    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:14.525222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.525205    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-31d2617b-dockercfg-k4vf5\""
Jul 28 17:37:14.535693 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.535669    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"]
Jul 28 17:37:14.537543 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.537522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef65452-9c3d-4578-abb0-f1967e41a90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r\" (UID: \"7ef65452-9c3d-4578-abb0-f1967e41a90e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:14.537627 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.537614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shlp\" (UniqueName: \"kubernetes.io/projected/7ef65452-9c3d-4578-abb0-f1967e41a90e-kube-api-access-9shlp\") pod \"payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r\" (UID: \"7ef65452-9c3d-4578-abb0-f1967e41a90e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:14.537671 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.537662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7t4b\" (UniqueName: \"kubernetes.io/projected/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-kube-api-access-v7t4b\") pod \"payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck\" (UID: \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\") " pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:14.537706 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.537691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-plugins-config-volume\") pod \"payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck\" (UID: \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\") " pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:14.638533 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.638503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shlp\" (UniqueName: \"kubernetes.io/projected/7ef65452-9c3d-4578-abb0-f1967e41a90e-kube-api-access-9shlp\") pod \"payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r\" (UID: \"7ef65452-9c3d-4578-abb0-f1967e41a90e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:14.638669 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.638547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7t4b\" (UniqueName: \"kubernetes.io/projected/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-kube-api-access-v7t4b\") pod \"payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck\" (UID: \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\") " pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:14.638669 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.638590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-plugins-config-volume\") pod \"payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck\" (UID: \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\") " pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:14.638669 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.638614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef65452-9c3d-4578-abb0-f1967e41a90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r\" (UID: \"7ef65452-9c3d-4578-abb0-f1967e41a90e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:14.639230 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.639212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-plugins-config-volume\") pod \"payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck\" (UID: \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\") " pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:14.639492 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.639474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef65452-9c3d-4578-abb0-f1967e41a90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r\" (UID: \"7ef65452-9c3d-4578-abb0-f1967e41a90e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:14.656871 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.656846    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7t4b\" (UniqueName: \"kubernetes.io/projected/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-kube-api-access-v7t4b\") pod \"payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck\" (UID: \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\") " pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:14.658194 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.658173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shlp\" (UniqueName: \"kubernetes.io/projected/7ef65452-9c3d-4578-abb0-f1967e41a90e-kube-api-access-9shlp\") pod \"payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r\" (UID: \"7ef65452-9c3d-4578-abb0-f1967e41a90e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:14.798336 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.798310    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:14.831195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.831166    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:14.887192 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.886670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w" event={"ID":"a51c837e-f224-4f7d-8f32-ac91de903074","Type":"ContainerStarted","Data":"07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921"}
Jul 28 17:37:14.887192 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.886731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w" event={"ID":"a51c837e-f224-4f7d-8f32-ac91de903074","Type":"ContainerStarted","Data":"dec38b22343682cb93b860f449928bdce937afc39516014fe3412e4da8f2122a"}
Jul 28 17:37:14.887192 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.886870    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:14.915681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.915424    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w" podStartSLOduration=1.304930402 podStartE2EDuration="1.915404816s" totalImagesPullingTime="610.474414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:14.913191638 +0000 UTC m=+1705.585519558" watchObservedRunningTime="2026-07-28 17:37:14.915404816 +0000 UTC m=+1705.587732737"
Jul 28 17:37:14.952982 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.952956    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"]
Jul 28 17:37:14.955408 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:37:14.955338    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef65452_9c3d_4578_abb0_f1967e41a90e.slice/crio-75312e44dca2484d0aca2f3c5b2d8230017415fefe17ae0887adf4c486c36cca WatchSource:0}: Error finding container 75312e44dca2484d0aca2f3c5b2d8230017415fefe17ae0887adf4c486c36cca: Status 404 returned error can't find the container with id 75312e44dca2484d0aca2f3c5b2d8230017415fefe17ae0887adf4c486c36cca
Jul 28 17:37:14.997508 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:14.997478    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"]
Jul 28 17:37:15.014435 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:37:15.014403    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c3ef43_72e0_4d1a_9a59_a77ae6eb40b2.slice/crio-17b4d2edec322f8a68625b503c3e1f3c580258ef40af338469e295d72fd77f61 WatchSource:0}: Error finding container 17b4d2edec322f8a68625b503c3e1f3c580258ef40af338469e295d72fd77f61: Status 404 returned error can't find the container with id 17b4d2edec322f8a68625b503c3e1f3c580258ef40af338469e295d72fd77f61
Jul 28 17:37:15.891895 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:15.891859    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck" event={"ID":"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2","Type":"ContainerStarted","Data":"64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c"}
Jul 28 17:37:15.892364 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:15.891913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck" event={"ID":"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2","Type":"ContainerStarted","Data":"17b4d2edec322f8a68625b503c3e1f3c580258ef40af338469e295d72fd77f61"}
Jul 28 17:37:15.892364 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:15.891943    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:15.893286 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:15.893264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r" event={"ID":"7ef65452-9c3d-4578-abb0-f1967e41a90e","Type":"ContainerStarted","Data":"f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73"}
Jul 28 17:37:15.893391 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:15.893302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r" event={"ID":"7ef65452-9c3d-4578-abb0-f1967e41a90e","Type":"ContainerStarted","Data":"75312e44dca2484d0aca2f3c5b2d8230017415fefe17ae0887adf4c486c36cca"}
Jul 28 17:37:15.911736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:15.911688    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck" podStartSLOduration=1.911674615 podStartE2EDuration="1.911674615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:15.909636017 +0000 UTC m=+1706.581963935" watchObservedRunningTime="2026-07-28 17:37:15.911674615 +0000 UTC m=+1706.584002534"
Jul 28 17:37:15.927221 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:15.927173    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r" podStartSLOduration=1.927165156 podStartE2EDuration="1.927165156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:15.92547884 +0000 UTC m=+1706.597806763" watchObservedRunningTime="2026-07-28 17:37:15.927165156 +0000 UTC m=+1706.599493075"
Jul 28 17:37:16.898254 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:16.898223    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:20.899116 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:20.899089    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:26.901042 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:26.901007    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:27.905614 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:27.905582    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:48.980706 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:48.980670    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"]
Jul 28 17:37:48.981270 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:48.980928    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w" podUID="a51c837e-f224-4f7d-8f32-ac91de903074" containerName="maas-api" containerID="cri-o://07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921" gracePeriod=30
Jul 28 17:37:49.043177 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.043144    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"]
Jul 28 17:37:49.043465 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.043416    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck" podUID="c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2" containerName="payload-processing" containerID="cri-o://64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c" gracePeriod=30
Jul 28 17:37:49.049363 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.049217    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"]
Jul 28 17:37:49.049546 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.049447    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r" podUID="7ef65452-9c3d-4578-abb0-f1967e41a90e" containerName="payload-pre-processing" containerID="cri-o://f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73" gracePeriod=30
Jul 28 17:37:49.297733 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.297702    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:49.319838 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.319812    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:49.336292 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.336268    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:49.468480 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51c837e-f224-4f7d-8f32-ac91de903074-maas-api-tls\") pod \"a51c837e-f224-4f7d-8f32-ac91de903074\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") "
Jul 28 17:37:49.468651 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468535    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51c837e-f224-4f7d-8f32-ac91de903074-openshift-service-ca\") pod \"a51c837e-f224-4f7d-8f32-ac91de903074\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") "
Jul 28 17:37:49.468651 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468569    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef65452-9c3d-4578-abb0-f1967e41a90e-plugins-config-volume\") pod \"7ef65452-9c3d-4578-abb0-f1967e41a90e\" (UID: \"7ef65452-9c3d-4578-abb0-f1967e41a90e\") "
Jul 28 17:37:49.468651 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468595    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7t4b\" (UniqueName: \"kubernetes.io/projected/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-kube-api-access-v7t4b\") pod \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\" (UID: \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\") "
Jul 28 17:37:49.468651 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468621    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppjnf\" (UniqueName: \"kubernetes.io/projected/a51c837e-f224-4f7d-8f32-ac91de903074-kube-api-access-ppjnf\") pod \"a51c837e-f224-4f7d-8f32-ac91de903074\" (UID: \"a51c837e-f224-4f7d-8f32-ac91de903074\") "
Jul 28 17:37:49.468862 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468665    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-plugins-config-volume\") pod \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\" (UID: \"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2\") "
Jul 28 17:37:49.468862 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468721    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9shlp\" (UniqueName: \"kubernetes.io/projected/7ef65452-9c3d-4578-abb0-f1967e41a90e-kube-api-access-9shlp\") pod \"7ef65452-9c3d-4578-abb0-f1967e41a90e\" (UID: \"7ef65452-9c3d-4578-abb0-f1967e41a90e\") "
Jul 28 17:37:49.468984 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468960    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51c837e-f224-4f7d-8f32-ac91de903074-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a51c837e-f224-4f7d-8f32-ac91de903074" (UID: "a51c837e-f224-4f7d-8f32-ac91de903074"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:49.469157 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.468984    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef65452-9c3d-4578-abb0-f1967e41a90e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ef65452-9c3d-4578-abb0-f1967e41a90e" (UID: "7ef65452-9c3d-4578-abb0-f1967e41a90e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:49.469225 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.469199    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2" (UID: "c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:49.470428 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.470405    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51c837e-f224-4f7d-8f32-ac91de903074-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a51c837e-f224-4f7d-8f32-ac91de903074" (UID: "a51c837e-f224-4f7d-8f32-ac91de903074"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:49.470773 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.470757    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-kube-api-access-v7t4b" (OuterVolumeSpecName: "kube-api-access-v7t4b") pod "c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2" (UID: "c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2"). InnerVolumeSpecName "kube-api-access-v7t4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:49.471149 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.471129    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef65452-9c3d-4578-abb0-f1967e41a90e-kube-api-access-9shlp" (OuterVolumeSpecName: "kube-api-access-9shlp") pod "7ef65452-9c3d-4578-abb0-f1967e41a90e" (UID: "7ef65452-9c3d-4578-abb0-f1967e41a90e"). InnerVolumeSpecName "kube-api-access-9shlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:49.471197 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.471142    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51c837e-f224-4f7d-8f32-ac91de903074-kube-api-access-ppjnf" (OuterVolumeSpecName: "kube-api-access-ppjnf") pod "a51c837e-f224-4f7d-8f32-ac91de903074" (UID: "a51c837e-f224-4f7d-8f32-ac91de903074"). InnerVolumeSpecName "kube-api-access-ppjnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:49.569727 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.569693    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef65452-9c3d-4578-abb0-f1967e41a90e-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:49.569727 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.569727    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7t4b\" (UniqueName: \"kubernetes.io/projected/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-kube-api-access-v7t4b\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:49.569727 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.569736    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjnf\" (UniqueName: \"kubernetes.io/projected/a51c837e-f224-4f7d-8f32-ac91de903074-kube-api-access-ppjnf\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:49.569976 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.569745    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:49.569976 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.569754    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9shlp\" (UniqueName: \"kubernetes.io/projected/7ef65452-9c3d-4578-abb0-f1967e41a90e-kube-api-access-9shlp\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:49.569976 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.569766    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51c837e-f224-4f7d-8f32-ac91de903074-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:49.569976 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:49.569774    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51c837e-f224-4f7d-8f32-ac91de903074-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:37:50.060908 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.060871    2560 generic.go:356] "Generic (PLEG): container finished" podID="c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2" containerID="64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c" exitCode=0
Jul 28 17:37:50.061395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.060914    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"
Jul 28 17:37:50.061395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.060954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck" event={"ID":"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2","Type":"ContainerDied","Data":"64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c"}
Jul 28 17:37:50.061395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.061005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck" event={"ID":"c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2","Type":"ContainerDied","Data":"17b4d2edec322f8a68625b503c3e1f3c580258ef40af338469e295d72fd77f61"}
Jul 28 17:37:50.061395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.061028    2560 scope.go:117] "RemoveContainer" containerID="64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c"
Jul 28 17:37:50.062456 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.062437    2560 generic.go:356] "Generic (PLEG): container finished" podID="a51c837e-f224-4f7d-8f32-ac91de903074" containerID="07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921" exitCode=0
Jul 28 17:37:50.062565 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.062493    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"
Jul 28 17:37:50.062565 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.062521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w" event={"ID":"a51c837e-f224-4f7d-8f32-ac91de903074","Type":"ContainerDied","Data":"07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921"}
Jul 28 17:37:50.062682 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.062563    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w" event={"ID":"a51c837e-f224-4f7d-8f32-ac91de903074","Type":"ContainerDied","Data":"dec38b22343682cb93b860f449928bdce937afc39516014fe3412e4da8f2122a"}
Jul 28 17:37:50.064078 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.064054    2560 generic.go:356] "Generic (PLEG): container finished" podID="7ef65452-9c3d-4578-abb0-f1967e41a90e" containerID="f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73" exitCode=0
Jul 28 17:37:50.064176 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.064116    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"
Jul 28 17:37:50.064176 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.064145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r" event={"ID":"7ef65452-9c3d-4578-abb0-f1967e41a90e","Type":"ContainerDied","Data":"f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73"}
Jul 28 17:37:50.064279 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.064185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r" event={"ID":"7ef65452-9c3d-4578-abb0-f1967e41a90e","Type":"ContainerDied","Data":"75312e44dca2484d0aca2f3c5b2d8230017415fefe17ae0887adf4c486c36cca"}
Jul 28 17:37:50.070468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.070406    2560 scope.go:117] "RemoveContainer" containerID="64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c"
Jul 28 17:37:50.070705 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:37:50.070687    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c\": container with ID starting with 64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c not found: ID does not exist" containerID="64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c"
Jul 28 17:37:50.070762 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.070712    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c"} err="failed to get container status \"64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c\": rpc error: code = NotFound desc = could not find container \"64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c\": container with ID starting with 64db4582b2a5ce1bf7ecf947ca614581e2dc94135217c3709e2f20a1a579298c not found: ID does not exist"
Jul 28 17:37:50.070762 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.070730    2560 scope.go:117] "RemoveContainer" containerID="07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921"
Jul 28 17:37:50.080496 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.080480    2560 scope.go:117] "RemoveContainer" containerID="07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921"
Jul 28 17:37:50.080759 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:37:50.080733    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921\": container with ID starting with 07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921 not found: ID does not exist" containerID="07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921"
Jul 28 17:37:50.080848 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.080758    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921"} err="failed to get container status \"07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921\": rpc error: code = NotFound desc = could not find container \"07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921\": container with ID starting with 07978ea7d3a1b9b9aa3a167cb02a6cc7c691b3ae6af81672b8a607f25554a921 not found: ID does not exist"
Jul 28 17:37:50.080848 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.080776    2560 scope.go:117] "RemoveContainer" containerID="f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73"
Jul 28 17:37:50.087212 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.087191    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"]
Jul 28 17:37:50.088376 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.088362    2560 scope.go:117] "RemoveContainer" containerID="f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73"
Jul 28 17:37:50.088619 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:37:50.088602    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73\": container with ID starting with f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73 not found: ID does not exist" containerID="f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73"
Jul 28 17:37:50.088671 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.088625    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73"} err="failed to get container status \"f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73\": rpc error: code = NotFound desc = could not find container \"f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73\": container with ID starting with f9fc51d4d2846f96c7c4870aeaf4af8bb70af7093c638f9df2974adcf582ba73 not found: ID does not exist"
Jul 28 17:37:50.093478 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.093457    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31d2617b-7cff4d86b4-c6w5r"]
Jul 28 17:37:50.104557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.104536    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"]
Jul 28 17:37:50.109210 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.109187    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31d2617b-7f4c8f6b89-m8p8w"]
Jul 28 17:37:50.122436 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.122417    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"]
Jul 28 17:37:50.124948 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:50.124928    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31d2617b-6cb4df986-ndrck"]
Jul 28 17:37:51.873892 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:51.873857    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef65452-9c3d-4578-abb0-f1967e41a90e" path="/var/lib/kubelet/pods/7ef65452-9c3d-4578-abb0-f1967e41a90e/volumes"
Jul 28 17:37:51.874252 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:51.874225    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51c837e-f224-4f7d-8f32-ac91de903074" path="/var/lib/kubelet/pods/a51c837e-f224-4f7d-8f32-ac91de903074/volumes"
Jul 28 17:37:51.874543 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:37:51.874529    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2" path="/var/lib/kubelet/pods/c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2/volumes"
Jul 28 17:38:49.876425 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:38:49.876329    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:38:49.880257 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:38:49.880237    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:39:38.631573 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.631539    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"]
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.631899    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2" containerName="payload-processing"
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.631910    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2" containerName="payload-processing"
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.631929    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51c837e-f224-4f7d-8f32-ac91de903074" containerName="maas-api"
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.631935    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51c837e-f224-4f7d-8f32-ac91de903074" containerName="maas-api"
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.631949    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef65452-9c3d-4578-abb0-f1967e41a90e" containerName="payload-pre-processing"
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.631954    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef65452-9c3d-4578-abb0-f1967e41a90e" containerName="payload-pre-processing"
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.632009    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef65452-9c3d-4578-abb0-f1967e41a90e" containerName="payload-pre-processing"
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.632018    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c3ef43-72e0-4d1a-9a59-a77ae6eb40b2" containerName="payload-processing"
Jul 28 17:39:38.632141 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.632025    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51c837e-f224-4f7d-8f32-ac91de903074" containerName="maas-api"
Jul 28 17:39:38.635332 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.635311    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:38.638124 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.638082    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:39:38.638316 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.638297    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d9be9a39-serving-cert\""
Jul 28 17:39:38.638427 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.638338    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:39:38.638753 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.638736    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:39:38.647485 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.647457    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"]
Jul 28 17:39:38.718518 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.718483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrc8\" (UniqueName: \"kubernetes.io/projected/ec48b595-78bd-4324-9dbe-b6d2cf105a63-kube-api-access-fhrc8\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:38.718672 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.718544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec48b595-78bd-4324-9dbe-b6d2cf105a63-openshift-service-ca\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:38.718672 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.718565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:38.819660 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.819628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrc8\" (UniqueName: \"kubernetes.io/projected/ec48b595-78bd-4324-9dbe-b6d2cf105a63-kube-api-access-fhrc8\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:38.819776 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.819669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec48b595-78bd-4324-9dbe-b6d2cf105a63-openshift-service-ca\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:38.819776 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.819688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:38.819847 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:39:38.819791    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-serving-cert: secret "maas-api-e2e-mt-d9be9a39-serving-cert" not found
Jul 28 17:39:38.819880 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:39:38.819847    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls podName:ec48b595-78bd-4324-9dbe-b6d2cf105a63 nodeName:}" failed. No retries permitted until 2026-07-28 17:39:39.319828749 +0000 UTC m=+1849.992156668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls") pod "maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c" (UID: "ec48b595-78bd-4324-9dbe-b6d2cf105a63") : secret "maas-api-e2e-mt-d9be9a39-serving-cert" not found
Jul 28 17:39:38.820214 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.820197    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec48b595-78bd-4324-9dbe-b6d2cf105a63-openshift-service-ca\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:38.835088 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:38.835063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrc8\" (UniqueName: \"kubernetes.io/projected/ec48b595-78bd-4324-9dbe-b6d2cf105a63-kube-api-access-fhrc8\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:39.323846 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.323809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:39.326377 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.326327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls\") pod \"maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:39.546257 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.546227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:39.557508 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.557486    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"]
Jul 28 17:39:39.561177 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.561156    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:39.564357 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.564309    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d9be9a39\""
Jul 28 17:39:39.564532 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.564515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d9be9a39-dockercfg-78fts\""
Jul 28 17:39:39.572079 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.572049    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"]
Jul 28 17:39:39.626984 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.626951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kq6c\" (UniqueName: \"kubernetes.io/projected/2ff57f91-7594-409a-b0d7-439c0c489543-kube-api-access-7kq6c\") pod \"payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk\" (UID: \"2ff57f91-7594-409a-b0d7-439c0c489543\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:39.626984 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.626983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff57f91-7594-409a-b0d7-439c0c489543-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk\" (UID: \"2ff57f91-7594-409a-b0d7-439c0c489543\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:39.664060 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.664027    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"]
Jul 28 17:39:39.668422 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.668397    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:39.677535 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.677493    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"]
Jul 28 17:39:39.683183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.683164    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"]
Jul 28 17:39:39.684931 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:39:39.684909    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec48b595_78bd_4324_9dbe_b6d2cf105a63.slice/crio-58560a3cbd726746bc174bdf540d8892acb52c55d3e06345e75a77aab4aefa78 WatchSource:0}: Error finding container 58560a3cbd726746bc174bdf540d8892acb52c55d3e06345e75a77aab4aefa78: Status 404 returned error can't find the container with id 58560a3cbd726746bc174bdf540d8892acb52c55d3e06345e75a77aab4aefa78
Jul 28 17:39:39.728021 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.727993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7bcd54-d7f4-4952-929f-28936adce048-plugins-config-volume\") pod \"payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42\" (UID: \"3b7bcd54-d7f4-4952-929f-28936adce048\") " pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:39.728133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.728054    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kq6c\" (UniqueName: \"kubernetes.io/projected/2ff57f91-7594-409a-b0d7-439c0c489543-kube-api-access-7kq6c\") pod \"payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk\" (UID: \"2ff57f91-7594-409a-b0d7-439c0c489543\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:39.728133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.728074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff57f91-7594-409a-b0d7-439c0c489543-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk\" (UID: \"2ff57f91-7594-409a-b0d7-439c0c489543\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:39.728242 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.728212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zf26\" (UniqueName: \"kubernetes.io/projected/3b7bcd54-d7f4-4952-929f-28936adce048-kube-api-access-6zf26\") pod \"payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42\" (UID: \"3b7bcd54-d7f4-4952-929f-28936adce048\") " pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:39.728709 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.728692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff57f91-7594-409a-b0d7-439c0c489543-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk\" (UID: \"2ff57f91-7594-409a-b0d7-439c0c489543\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:39.738288 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.738268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kq6c\" (UniqueName: \"kubernetes.io/projected/2ff57f91-7594-409a-b0d7-439c0c489543-kube-api-access-7kq6c\") pod \"payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk\" (UID: \"2ff57f91-7594-409a-b0d7-439c0c489543\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:39.828901 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.828830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zf26\" (UniqueName: \"kubernetes.io/projected/3b7bcd54-d7f4-4952-929f-28936adce048-kube-api-access-6zf26\") pod \"payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42\" (UID: \"3b7bcd54-d7f4-4952-929f-28936adce048\") " pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:39.829020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.828909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7bcd54-d7f4-4952-929f-28936adce048-plugins-config-volume\") pod \"payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42\" (UID: \"3b7bcd54-d7f4-4952-929f-28936adce048\") " pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:39.829594 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.829577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7bcd54-d7f4-4952-929f-28936adce048-plugins-config-volume\") pod \"payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42\" (UID: \"3b7bcd54-d7f4-4952-929f-28936adce048\") " pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:39.837583 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.837560    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zf26\" (UniqueName: \"kubernetes.io/projected/3b7bcd54-d7f4-4952-929f-28936adce048-kube-api-access-6zf26\") pod \"payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42\" (UID: \"3b7bcd54-d7f4-4952-929f-28936adce048\") " pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:39.880701 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.880677    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:39.980692 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:39.980662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:40.014306 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.014248    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"]
Jul 28 17:39:40.016885 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:39:40.016848    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff57f91_7594_409a_b0d7_439c0c489543.slice/crio-fc63973ba869eba102519932c086e50025e1077cdebfb64a2fb72db2d0846b7c WatchSource:0}: Error finding container fc63973ba869eba102519932c086e50025e1077cdebfb64a2fb72db2d0846b7c: Status 404 returned error can't find the container with id fc63973ba869eba102519932c086e50025e1077cdebfb64a2fb72db2d0846b7c
Jul 28 17:39:40.119754 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.119729    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"]
Jul 28 17:39:40.123268 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:39:40.123245    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7bcd54_d7f4_4952_929f_28936adce048.slice/crio-bd769ca6374d3c3ef05505a2add5d8048bdd5cf99d278e860ba078800f6e8202 WatchSource:0}: Error finding container bd769ca6374d3c3ef05505a2add5d8048bdd5cf99d278e860ba078800f6e8202: Status 404 returned error can't find the container with id bd769ca6374d3c3ef05505a2add5d8048bdd5cf99d278e860ba078800f6e8202
Jul 28 17:39:40.570882 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.570846    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c" event={"ID":"ec48b595-78bd-4324-9dbe-b6d2cf105a63","Type":"ContainerStarted","Data":"782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7"}
Jul 28 17:39:40.571078 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.570902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c" event={"ID":"ec48b595-78bd-4324-9dbe-b6d2cf105a63","Type":"ContainerStarted","Data":"58560a3cbd726746bc174bdf540d8892acb52c55d3e06345e75a77aab4aefa78"}
Jul 28 17:39:40.571078 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.570926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:40.572406 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.572375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk" event={"ID":"2ff57f91-7594-409a-b0d7-439c0c489543","Type":"ContainerStarted","Data":"1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2"}
Jul 28 17:39:40.572523 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.572428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk" event={"ID":"2ff57f91-7594-409a-b0d7-439c0c489543","Type":"ContainerStarted","Data":"fc63973ba869eba102519932c086e50025e1077cdebfb64a2fb72db2d0846b7c"}
Jul 28 17:39:40.572523 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.572480    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:39:40.573887 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.573868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42" event={"ID":"3b7bcd54-d7f4-4952-929f-28936adce048","Type":"ContainerStarted","Data":"da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b"}
Jul 28 17:39:40.574002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.573901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42" event={"ID":"3b7bcd54-d7f4-4952-929f-28936adce048","Type":"ContainerStarted","Data":"bd769ca6374d3c3ef05505a2add5d8048bdd5cf99d278e860ba078800f6e8202"}
Jul 28 17:39:40.574002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.573920    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:39:40.588698 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.588660    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c" podStartSLOduration=2.165034076 podStartE2EDuration="2.588648262s" totalImagesPullingTime="423.614186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:40.587127466 +0000 UTC m=+1851.259455388" watchObservedRunningTime="2026-07-28 17:39:40.588648262 +0000 UTC m=+1851.260976181"
Jul 28 17:39:40.603798 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.603759    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk" podStartSLOduration=1.6037466299999998 podStartE2EDuration="1.60374663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:40.602667511 +0000 UTC m=+1851.274995430" watchObservedRunningTime="2026-07-28 17:39:40.60374663 +0000 UTC m=+1851.276074549"
Jul 28 17:39:40.620701 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:40.620658    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42" podStartSLOduration=1.620645912 podStartE2EDuration="1.620645912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:40.619080423 +0000 UTC m=+1851.291408345" watchObservedRunningTime="2026-07-28 17:39:40.620645912 +0000 UTC m=+1851.292973831"
Jul 28 17:39:46.583870 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:46.583837    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:39:51.580455 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:51.580401    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk" podUID="2ff57f91-7594-409a-b0d7-439c0c489543" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.43:9004: connect: connection refused"
Jul 28 17:39:51.581030 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:39:51.581011    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:40:01.099147 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.099111    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"]
Jul 28 17:40:01.099546 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.099293    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c" podUID="ec48b595-78bd-4324-9dbe-b6d2cf105a63" containerName="maas-api" containerID="cri-o://782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7" gracePeriod=30
Jul 28 17:40:01.160499 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.160464    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"]
Jul 28 17:40:01.160713 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.160688    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42" podUID="3b7bcd54-d7f4-4952-929f-28936adce048" containerName="payload-processing" containerID="cri-o://da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b" gracePeriod=30
Jul 28 17:40:01.176706 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.176677    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"]
Jul 28 17:40:01.176942 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.176907    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk" podUID="2ff57f91-7594-409a-b0d7-439c0c489543" containerName="payload-pre-processing" containerID="cri-o://1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2" gracePeriod=30
Jul 28 17:40:01.177620 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.177596    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:40:01.434076 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.434052    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:40:01.443409 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.443385    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:40:01.450523 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.450506    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:40:01.531310 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531282    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kq6c\" (UniqueName: \"kubernetes.io/projected/2ff57f91-7594-409a-b0d7-439c0c489543-kube-api-access-7kq6c\") pod \"2ff57f91-7594-409a-b0d7-439c0c489543\" (UID: \"2ff57f91-7594-409a-b0d7-439c0c489543\") "
Jul 28 17:40:01.531310 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7bcd54-d7f4-4952-929f-28936adce048-plugins-config-volume\") pod \"3b7bcd54-d7f4-4952-929f-28936adce048\" (UID: \"3b7bcd54-d7f4-4952-929f-28936adce048\") "
Jul 28 17:40:01.531493 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531399    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhrc8\" (UniqueName: \"kubernetes.io/projected/ec48b595-78bd-4324-9dbe-b6d2cf105a63-kube-api-access-fhrc8\") pod \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") "
Jul 28 17:40:01.531493 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531423    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff57f91-7594-409a-b0d7-439c0c489543-plugins-config-volume\") pod \"2ff57f91-7594-409a-b0d7-439c0c489543\" (UID: \"2ff57f91-7594-409a-b0d7-439c0c489543\") "
Jul 28 17:40:01.531493 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531470    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec48b595-78bd-4324-9dbe-b6d2cf105a63-openshift-service-ca\") pod \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") "
Jul 28 17:40:01.531648 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531523    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zf26\" (UniqueName: \"kubernetes.io/projected/3b7bcd54-d7f4-4952-929f-28936adce048-kube-api-access-6zf26\") pod \"3b7bcd54-d7f4-4952-929f-28936adce048\" (UID: \"3b7bcd54-d7f4-4952-929f-28936adce048\") "
Jul 28 17:40:01.531648 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls\") pod \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\" (UID: \"ec48b595-78bd-4324-9dbe-b6d2cf105a63\") "
Jul 28 17:40:01.531833 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531804    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff57f91-7594-409a-b0d7-439c0c489543-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ff57f91-7594-409a-b0d7-439c0c489543" (UID: "2ff57f91-7594-409a-b0d7-439c0c489543"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:01.532074 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.531854    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec48b595-78bd-4324-9dbe-b6d2cf105a63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec48b595-78bd-4324-9dbe-b6d2cf105a63" (UID: "ec48b595-78bd-4324-9dbe-b6d2cf105a63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:01.532151 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.532081    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7bcd54-d7f4-4952-929f-28936adce048-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b7bcd54-d7f4-4952-929f-28936adce048" (UID: "3b7bcd54-d7f4-4952-929f-28936adce048"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:01.533517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.533493    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec48b595-78bd-4324-9dbe-b6d2cf105a63-kube-api-access-fhrc8" (OuterVolumeSpecName: "kube-api-access-fhrc8") pod "ec48b595-78bd-4324-9dbe-b6d2cf105a63" (UID: "ec48b595-78bd-4324-9dbe-b6d2cf105a63"). InnerVolumeSpecName "kube-api-access-fhrc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:01.533606 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.533585    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec48b595-78bd-4324-9dbe-b6d2cf105a63" (UID: "ec48b595-78bd-4324-9dbe-b6d2cf105a63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:01.533888 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.533869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7bcd54-d7f4-4952-929f-28936adce048-kube-api-access-6zf26" (OuterVolumeSpecName: "kube-api-access-6zf26") pod "3b7bcd54-d7f4-4952-929f-28936adce048" (UID: "3b7bcd54-d7f4-4952-929f-28936adce048"). InnerVolumeSpecName "kube-api-access-6zf26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:01.533888 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.533872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff57f91-7594-409a-b0d7-439c0c489543-kube-api-access-7kq6c" (OuterVolumeSpecName: "kube-api-access-7kq6c") pod "2ff57f91-7594-409a-b0d7-439c0c489543" (UID: "2ff57f91-7594-409a-b0d7-439c0c489543"). InnerVolumeSpecName "kube-api-access-7kq6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:01.633038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.632977    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec48b595-78bd-4324-9dbe-b6d2cf105a63-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:40:01.633038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.632999    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kq6c\" (UniqueName: \"kubernetes.io/projected/2ff57f91-7594-409a-b0d7-439c0c489543-kube-api-access-7kq6c\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:40:01.633038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.633009    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b7bcd54-d7f4-4952-929f-28936adce048-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:40:01.633038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.633018    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhrc8\" (UniqueName: \"kubernetes.io/projected/ec48b595-78bd-4324-9dbe-b6d2cf105a63-kube-api-access-fhrc8\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:40:01.633038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.633026    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff57f91-7594-409a-b0d7-439c0c489543-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:40:01.633038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.633034    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec48b595-78bd-4324-9dbe-b6d2cf105a63-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:40:01.633280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.633043    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zf26\" (UniqueName: \"kubernetes.io/projected/3b7bcd54-d7f4-4952-929f-28936adce048-kube-api-access-6zf26\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:40:01.674085 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.674057    2560 generic.go:356] "Generic (PLEG): container finished" podID="2ff57f91-7594-409a-b0d7-439c0c489543" containerID="1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2" exitCode=0
Jul 28 17:40:01.674195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.674116    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"
Jul 28 17:40:01.674195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.674151    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk" event={"ID":"2ff57f91-7594-409a-b0d7-439c0c489543","Type":"ContainerDied","Data":"1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2"}
Jul 28 17:40:01.674304 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.674236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk" event={"ID":"2ff57f91-7594-409a-b0d7-439c0c489543","Type":"ContainerDied","Data":"fc63973ba869eba102519932c086e50025e1077cdebfb64a2fb72db2d0846b7c"}
Jul 28 17:40:01.674304 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.674269    2560 scope.go:117] "RemoveContainer" containerID="1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2"
Jul 28 17:40:01.675381 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.675363    2560 generic.go:356] "Generic (PLEG): container finished" podID="3b7bcd54-d7f4-4952-929f-28936adce048" containerID="da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b" exitCode=0
Jul 28 17:40:01.675461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.675435    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"
Jul 28 17:40:01.675524 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.675461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42" event={"ID":"3b7bcd54-d7f4-4952-929f-28936adce048","Type":"ContainerDied","Data":"da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b"}
Jul 28 17:40:01.675566 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.675520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42" event={"ID":"3b7bcd54-d7f4-4952-929f-28936adce048","Type":"ContainerDied","Data":"bd769ca6374d3c3ef05505a2add5d8048bdd5cf99d278e860ba078800f6e8202"}
Jul 28 17:40:01.676679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.676654    2560 generic.go:356] "Generic (PLEG): container finished" podID="ec48b595-78bd-4324-9dbe-b6d2cf105a63" containerID="782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7" exitCode=0
Jul 28 17:40:01.676808 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.676691    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c" event={"ID":"ec48b595-78bd-4324-9dbe-b6d2cf105a63","Type":"ContainerDied","Data":"782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7"}
Jul 28 17:40:01.676808 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.676708    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"
Jul 28 17:40:01.676808 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.676765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c" event={"ID":"ec48b595-78bd-4324-9dbe-b6d2cf105a63","Type":"ContainerDied","Data":"58560a3cbd726746bc174bdf540d8892acb52c55d3e06345e75a77aab4aefa78"}
Jul 28 17:40:01.684252 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.684210    2560 scope.go:117] "RemoveContainer" containerID="1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2"
Jul 28 17:40:01.684566 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:40:01.684539    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2\": container with ID starting with 1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2 not found: ID does not exist" containerID="1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2"
Jul 28 17:40:01.684668 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.684574    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2"} err="failed to get container status \"1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2\": rpc error: code = NotFound desc = could not find container \"1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2\": container with ID starting with 1d6b25e0c99cd0561a9266865e8c1d9843fb6611f99fceba03c1d68cbb4a19b2 not found: ID does not exist"
Jul 28 17:40:01.684668 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.684597    2560 scope.go:117] "RemoveContainer" containerID="da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b"
Jul 28 17:40:01.692712 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.692696    2560 scope.go:117] "RemoveContainer" containerID="da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b"
Jul 28 17:40:01.692935 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:40:01.692915    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b\": container with ID starting with da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b not found: ID does not exist" containerID="da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b"
Jul 28 17:40:01.693003 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.692944    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b"} err="failed to get container status \"da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b\": rpc error: code = NotFound desc = could not find container \"da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b\": container with ID starting with da2f398c58770ed00b782091d67cf3850fd653e2b0feba0766965e83f249656b not found: ID does not exist"
Jul 28 17:40:01.693003 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.692964    2560 scope.go:117] "RemoveContainer" containerID="782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7"
Jul 28 17:40:01.700094 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.700080    2560 scope.go:117] "RemoveContainer" containerID="782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7"
Jul 28 17:40:01.700354 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:40:01.700324    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7\": container with ID starting with 782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7 not found: ID does not exist" containerID="782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7"
Jul 28 17:40:01.700424 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.700405    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7"} err="failed to get container status \"782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7\": rpc error: code = NotFound desc = could not find container \"782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7\": container with ID starting with 782eb0588fe709371e6131b13a27049e99f3758eb9ac075a31a8e57e24bae4e7 not found: ID does not exist"
Jul 28 17:40:01.702812 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.702791    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"]
Jul 28 17:40:01.706911 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.706888    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9be9a39-848f67cf9b-tp7kk"]
Jul 28 17:40:01.717756 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.717738    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"]
Jul 28 17:40:01.722556 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.722538    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9be9a39-54f9f5c7c4-x5j42"]
Jul 28 17:40:01.733630 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.733600    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"]
Jul 28 17:40:01.737920 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.737904    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9be9a39-66bc6cfc7-fw82c"]
Jul 28 17:40:01.872567 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.872535    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff57f91-7594-409a-b0d7-439c0c489543" path="/var/lib/kubelet/pods/2ff57f91-7594-409a-b0d7-439c0c489543/volumes"
Jul 28 17:40:01.872932 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.872918    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b7bcd54-d7f4-4952-929f-28936adce048" path="/var/lib/kubelet/pods/3b7bcd54-d7f4-4952-929f-28936adce048/volumes"
Jul 28 17:40:01.873290 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:01.873276    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec48b595-78bd-4324-9dbe-b6d2cf105a63" path="/var/lib/kubelet/pods/ec48b595-78bd-4324-9dbe-b6d2cf105a63/volumes"
Jul 28 17:40:36.830376 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.830287    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"]
Jul 28 17:40:36.830861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.830846    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff57f91-7594-409a-b0d7-439c0c489543" containerName="payload-pre-processing"
Jul 28 17:40:36.830906 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.830865    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff57f91-7594-409a-b0d7-439c0c489543" containerName="payload-pre-processing"
Jul 28 17:40:36.830906 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.830880    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec48b595-78bd-4324-9dbe-b6d2cf105a63" containerName="maas-api"
Jul 28 17:40:36.830906 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.830890    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec48b595-78bd-4324-9dbe-b6d2cf105a63" containerName="maas-api"
Jul 28 17:40:36.830996 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.830922    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7bcd54-d7f4-4952-929f-28936adce048" containerName="payload-processing"
Jul 28 17:40:36.830996 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.830931    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7bcd54-d7f4-4952-929f-28936adce048" containerName="payload-processing"
Jul 28 17:40:36.831055 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.831009    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff57f91-7594-409a-b0d7-439c0c489543" containerName="payload-pre-processing"
Jul 28 17:40:36.831055 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.831022    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec48b595-78bd-4324-9dbe-b6d2cf105a63" containerName="maas-api"
Jul 28 17:40:36.831055 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.831036    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7bcd54-d7f4-4952-929f-28936adce048" containerName="payload-processing"
Jul 28 17:40:36.834622 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.834599    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:36.837650 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.837613    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:40:36.837789 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.837723    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4794d51a-serving-cert\""
Jul 28 17:40:36.838431 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.838413    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:40:36.838495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.838438    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:40:36.845917 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.845895    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"]
Jul 28 17:40:36.954754 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.954724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8sbm\" (UniqueName: \"kubernetes.io/projected/b42ae312-9026-4b68-8f00-619b799247d5-kube-api-access-t8sbm\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:36.954906 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.954761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ae312-9026-4b68-8f00-619b799247d5-openshift-service-ca\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:36.954906 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:36.954794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42ae312-9026-4b68-8f00-619b799247d5-maas-api-tls\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.056078 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.056056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8sbm\" (UniqueName: \"kubernetes.io/projected/b42ae312-9026-4b68-8f00-619b799247d5-kube-api-access-t8sbm\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.056208 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.056087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ae312-9026-4b68-8f00-619b799247d5-openshift-service-ca\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.056208 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.056117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42ae312-9026-4b68-8f00-619b799247d5-maas-api-tls\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.056728 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.056710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ae312-9026-4b68-8f00-619b799247d5-openshift-service-ca\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.058337 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.058322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42ae312-9026-4b68-8f00-619b799247d5-maas-api-tls\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.064200 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.064180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8sbm\" (UniqueName: \"kubernetes.io/projected/b42ae312-9026-4b68-8f00-619b799247d5-kube-api-access-t8sbm\") pod \"maas-api-e2e-mt-4794d51a-74df677776-qlq4p\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.144585 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.144515    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.276298 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.276275    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"]
Jul 28 17:40:37.278972 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:40:37.278946    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42ae312_9026_4b68_8f00_619b799247d5.slice/crio-374401b2ac6f94211d26a7069fe3e9fc6092a12e92dbda50792818266f2fc90f WatchSource:0}: Error finding container 374401b2ac6f94211d26a7069fe3e9fc6092a12e92dbda50792818266f2fc90f: Status 404 returned error can't find the container with id 374401b2ac6f94211d26a7069fe3e9fc6092a12e92dbda50792818266f2fc90f
Jul 28 17:40:37.280295 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.280275    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:40:37.551812 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.551780    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"]
Jul 28 17:40:37.556509 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.556491    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:37.559014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.558995    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4794d51a\""
Jul 28 17:40:37.567768 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.567748    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"]
Jul 28 17:40:37.594951 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.594926    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"]
Jul 28 17:40:37.598787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.598771    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:37.601810 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.601793    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4794d51a-dockercfg-zkwk2\""
Jul 28 17:40:37.609588 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.609569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"]
Jul 28 17:40:37.660353 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.660318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d243d0f-9903-4b0a-a56f-01714bfb29be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7\" (UID: \"1d243d0f-9903-4b0a-a56f-01714bfb29be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:37.660451 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.660430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp699\" (UniqueName: \"kubernetes.io/projected/1d243d0f-9903-4b0a-a56f-01714bfb29be-kube-api-access-zp699\") pod \"payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7\" (UID: \"1d243d0f-9903-4b0a-a56f-01714bfb29be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:37.761380 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.761323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d243d0f-9903-4b0a-a56f-01714bfb29be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7\" (UID: \"1d243d0f-9903-4b0a-a56f-01714bfb29be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:37.761380 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.761385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-plugins-config-volume\") pod \"payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d\" (UID: \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\") " pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:37.761601 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.761411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25jj\" (UniqueName: \"kubernetes.io/projected/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-kube-api-access-k25jj\") pod \"payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d\" (UID: \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\") " pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:37.761601 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.761444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp699\" (UniqueName: \"kubernetes.io/projected/1d243d0f-9903-4b0a-a56f-01714bfb29be-kube-api-access-zp699\") pod \"payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7\" (UID: \"1d243d0f-9903-4b0a-a56f-01714bfb29be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:37.762104 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.762078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d243d0f-9903-4b0a-a56f-01714bfb29be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7\" (UID: \"1d243d0f-9903-4b0a-a56f-01714bfb29be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:37.770231 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.770211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp699\" (UniqueName: \"kubernetes.io/projected/1d243d0f-9903-4b0a-a56f-01714bfb29be-kube-api-access-zp699\") pod \"payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7\" (UID: \"1d243d0f-9903-4b0a-a56f-01714bfb29be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:37.843543 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.843484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p" event={"ID":"b42ae312-9026-4b68-8f00-619b799247d5","Type":"ContainerStarted","Data":"2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947"}
Jul 28 17:40:37.843543 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.843530    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p" event={"ID":"b42ae312-9026-4b68-8f00-619b799247d5","Type":"ContainerStarted","Data":"374401b2ac6f94211d26a7069fe3e9fc6092a12e92dbda50792818266f2fc90f"}
Jul 28 17:40:37.843884 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.843615    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:37.862358 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.862322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-plugins-config-volume\") pod \"payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d\" (UID: \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\") " pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:37.862443 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.862378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25jj\" (UniqueName: \"kubernetes.io/projected/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-kube-api-access-k25jj\") pod \"payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d\" (UID: \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\") " pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:37.862959 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.862940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-plugins-config-volume\") pod \"payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d\" (UID: \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\") " pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:37.867316 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.867298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:37.873481 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.873438    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p" podStartSLOduration=1.533806417 podStartE2EDuration="1.873422838s" totalImagesPullingTime="339.616421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:37.870489184 +0000 UTC m=+1908.542817104" watchObservedRunningTime="2026-07-28 17:40:37.873422838 +0000 UTC m=+1908.545750764"
Jul 28 17:40:37.874237 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.874220    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25jj\" (UniqueName: \"kubernetes.io/projected/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-kube-api-access-k25jj\") pod \"payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d\" (UID: \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\") " pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:37.908361 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:37.908318    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:38.029983 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.029957    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"]
Jul 28 17:40:38.032305 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:40:38.032280    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d243d0f_9903_4b0a_a56f_01714bfb29be.slice/crio-58c9229996c070ca65afbbde45591dc211c0f42159ec003a89aecbb288145804 WatchSource:0}: Error finding container 58c9229996c070ca65afbbde45591dc211c0f42159ec003a89aecbb288145804: Status 404 returned error can't find the container with id 58c9229996c070ca65afbbde45591dc211c0f42159ec003a89aecbb288145804
Jul 28 17:40:38.049755 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.049734    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"]
Jul 28 17:40:38.050947 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:40:38.050913    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cc09d7_9bcd_4493_a44f_d58b48e8d56b.slice/crio-3b57582cb7d46c893688aaffcf7da11738fc44e0645709200d67a0cf79cb806b WatchSource:0}: Error finding container 3b57582cb7d46c893688aaffcf7da11738fc44e0645709200d67a0cf79cb806b: Status 404 returned error can't find the container with id 3b57582cb7d46c893688aaffcf7da11738fc44e0645709200d67a0cf79cb806b
Jul 28 17:40:38.849792 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.849756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d" event={"ID":"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b","Type":"ContainerStarted","Data":"a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20"}
Jul 28 17:40:38.850223 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.849806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d" event={"ID":"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b","Type":"ContainerStarted","Data":"3b57582cb7d46c893688aaffcf7da11738fc44e0645709200d67a0cf79cb806b"}
Jul 28 17:40:38.850223 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.849896    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:38.851273 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.851250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7" event={"ID":"1d243d0f-9903-4b0a-a56f-01714bfb29be","Type":"ContainerStarted","Data":"e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2"}
Jul 28 17:40:38.851396 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.851295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7" event={"ID":"1d243d0f-9903-4b0a-a56f-01714bfb29be","Type":"ContainerStarted","Data":"58c9229996c070ca65afbbde45591dc211c0f42159ec003a89aecbb288145804"}
Jul 28 17:40:38.851396 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.851326    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:38.868331 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.868292    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d" podStartSLOduration=1.868280202 podStartE2EDuration="1.868280202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:38.867519696 +0000 UTC m=+1909.539847619" watchObservedRunningTime="2026-07-28 17:40:38.868280202 +0000 UTC m=+1909.540608168"
Jul 28 17:40:38.886506 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:38.886473    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7" podStartSLOduration=1.886461014 podStartE2EDuration="1.886461014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:38.884886493 +0000 UTC m=+1909.557214423" watchObservedRunningTime="2026-07-28 17:40:38.886461014 +0000 UTC m=+1909.558788979"
Jul 28 17:40:43.856280 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:43.856245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:40:49.858827 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:49.858794    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:40:49.859204 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:49.858877    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:40:53.931877 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:53.931840    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"]
Jul 28 17:40:53.935289 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:53.935273    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:53.938553 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:53.938521    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4794d51a\"/\"default-dockercfg-h22w6\""
Jul 28 17:40:53.938661 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:53.938572    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4794d51a\"/\"openshift-service-ca.crt\""
Jul 28 17:40:53.938661 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:53.938591    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4794d51a\"/\"kube-root-ca.crt\""
Jul 28 17:40:53.938661 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:53.938597    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4794d51a\"/\"e2e-lifecycle-model-4794d51a-kserve-self-signed-certs\""
Jul 28 17:40:53.947553 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:53.947531    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"]
Jul 28 17:40:54.001229 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.001199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tmp-dir\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.001365 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.001267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbk2j\" (UniqueName: \"kubernetes.io/projected/92a0ff7a-843c-48e1-8b9e-3c57dc772830-kube-api-access-xbk2j\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.001365 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.001338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-dshm\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.001460 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.001387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-home\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.001501 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.001479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tls-certs\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.001535 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.001517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-model-cache\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102327 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tls-certs\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-model-cache\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tmp-dir\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbk2j\" (UniqueName: \"kubernetes.io/projected/92a0ff7a-843c-48e1-8b9e-3c57dc772830-kube-api-access-xbk2j\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-dshm\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102663 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-home\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102815 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-model-cache\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102925 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tmp-dir\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.102925 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.102867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-home\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.104631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.104604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-dshm\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.104743 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.104726    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tls-certs\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.119252 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.119234    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbk2j\" (UniqueName: \"kubernetes.io/projected/92a0ff7a-843c-48e1-8b9e-3c57dc772830-kube-api-access-xbk2j\") pod \"e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") " pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.244146 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.244089    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:40:54.372568 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.372545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"]
Jul 28 17:40:54.374541 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:40:54.374517    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92a0ff7a_843c_48e1_8b9e_3c57dc772830.slice/crio-6c138c2fb0bbd6e2158857b7a1d8a6763994b185c1b2dfdf9e82ee6ee56f03fb WatchSource:0}: Error finding container 6c138c2fb0bbd6e2158857b7a1d8a6763994b185c1b2dfdf9e82ee6ee56f03fb: Status 404 returned error can't find the container with id 6c138c2fb0bbd6e2158857b7a1d8a6763994b185c1b2dfdf9e82ee6ee56f03fb
Jul 28 17:40:54.945946 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.945913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9" event={"ID":"92a0ff7a-843c-48e1-8b9e-3c57dc772830","Type":"ContainerStarted","Data":"ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3"}
Jul 28 17:40:54.946327 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.945955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9" event={"ID":"92a0ff7a-843c-48e1-8b9e-3c57dc772830","Type":"ContainerStarted","Data":"6c138c2fb0bbd6e2158857b7a1d8a6763994b185c1b2dfdf9e82ee6ee56f03fb"}
Jul 28 17:40:54.967047 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:40:54.967002    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9" podStartSLOduration=1.9669892949999999 podStartE2EDuration="1.966989295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:54.965763975 +0000 UTC m=+1925.638091894" watchObservedRunningTime="2026-07-28 17:40:54.966989295 +0000 UTC m=+1925.639317235"
Jul 28 17:41:04.244970 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:04.244936    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:41:04.245390 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:04.244983    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:41:04.256900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:04.256875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:41:05.005191 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:05.005162    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:41:46.779861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:46.779827    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"]
Jul 28 17:41:46.780282 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:46.780080    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p" podUID="b42ae312-9026-4b68-8f00-619b799247d5" containerName="maas-api" containerID="cri-o://2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947" gracePeriod=30
Jul 28 17:41:46.840427 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:46.840394    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"]
Jul 28 17:41:46.840641 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:46.840617    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d" podUID="e3cc09d7-9bcd-4493-a44f-d58b48e8d56b" containerName="payload-processing" containerID="cri-o://a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20" gracePeriod=30
Jul 28 17:41:46.848472 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:46.848449    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"]
Jul 28 17:41:46.848694 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:46.848667    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7" podUID="1d243d0f-9903-4b0a-a56f-01714bfb29be" containerName="payload-pre-processing" containerID="cri-o://e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2" gracePeriod=30
Jul 28 17:41:47.095369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.095330    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:41:47.110792 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.110768    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:41:47.148524 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.148501    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:41:47.173140 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.173108    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8sbm\" (UniqueName: \"kubernetes.io/projected/b42ae312-9026-4b68-8f00-619b799247d5-kube-api-access-t8sbm\") pod \"b42ae312-9026-4b68-8f00-619b799247d5\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") "
Jul 28 17:41:47.173315 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.173210    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42ae312-9026-4b68-8f00-619b799247d5-maas-api-tls\") pod \"b42ae312-9026-4b68-8f00-619b799247d5\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") "
Jul 28 17:41:47.173315 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.173241    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-plugins-config-volume\") pod \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\" (UID: \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\") "
Jul 28 17:41:47.173315 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.173262    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k25jj\" (UniqueName: \"kubernetes.io/projected/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-kube-api-access-k25jj\") pod \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\" (UID: \"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b\") "
Jul 28 17:41:47.173315 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.173292    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp699\" (UniqueName: \"kubernetes.io/projected/1d243d0f-9903-4b0a-a56f-01714bfb29be-kube-api-access-zp699\") pod \"1d243d0f-9903-4b0a-a56f-01714bfb29be\" (UID: \"1d243d0f-9903-4b0a-a56f-01714bfb29be\") "
Jul 28 17:41:47.173579 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.173434    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ae312-9026-4b68-8f00-619b799247d5-openshift-service-ca\") pod \"b42ae312-9026-4b68-8f00-619b799247d5\" (UID: \"b42ae312-9026-4b68-8f00-619b799247d5\") "
Jul 28 17:41:47.173579 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.173460    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d243d0f-9903-4b0a-a56f-01714bfb29be-plugins-config-volume\") pod \"1d243d0f-9903-4b0a-a56f-01714bfb29be\" (UID: \"1d243d0f-9903-4b0a-a56f-01714bfb29be\") "
Jul 28 17:41:47.174084 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.174052    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d243d0f-9903-4b0a-a56f-01714bfb29be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d243d0f-9903-4b0a-a56f-01714bfb29be" (UID: "1d243d0f-9903-4b0a-a56f-01714bfb29be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:47.174084 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.174059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42ae312-9026-4b68-8f00-619b799247d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b42ae312-9026-4b68-8f00-619b799247d5" (UID: "b42ae312-9026-4b68-8f00-619b799247d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:47.174258 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.174235    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3cc09d7-9bcd-4493-a44f-d58b48e8d56b" (UID: "e3cc09d7-9bcd-4493-a44f-d58b48e8d56b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:47.176056 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.176000    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42ae312-9026-4b68-8f00-619b799247d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b42ae312-9026-4b68-8f00-619b799247d5" (UID: "b42ae312-9026-4b68-8f00-619b799247d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:47.176302 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.176261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-kube-api-access-k25jj" (OuterVolumeSpecName: "kube-api-access-k25jj") pod "e3cc09d7-9bcd-4493-a44f-d58b48e8d56b" (UID: "e3cc09d7-9bcd-4493-a44f-d58b48e8d56b"). InnerVolumeSpecName "kube-api-access-k25jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:47.176427 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.176401    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42ae312-9026-4b68-8f00-619b799247d5-kube-api-access-t8sbm" (OuterVolumeSpecName: "kube-api-access-t8sbm") pod "b42ae312-9026-4b68-8f00-619b799247d5" (UID: "b42ae312-9026-4b68-8f00-619b799247d5"). InnerVolumeSpecName "kube-api-access-t8sbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:47.176626 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.176604    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d243d0f-9903-4b0a-a56f-01714bfb29be-kube-api-access-zp699" (OuterVolumeSpecName: "kube-api-access-zp699") pod "1d243d0f-9903-4b0a-a56f-01714bfb29be" (UID: "1d243d0f-9903-4b0a-a56f-01714bfb29be"). InnerVolumeSpecName "kube-api-access-zp699". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:47.202138 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.202116    2560 generic.go:356] "Generic (PLEG): container finished" podID="b42ae312-9026-4b68-8f00-619b799247d5" containerID="2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947" exitCode=0
Jul 28 17:41:47.202243 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.202172    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"
Jul 28 17:41:47.202243 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.202189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p" event={"ID":"b42ae312-9026-4b68-8f00-619b799247d5","Type":"ContainerDied","Data":"2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947"}
Jul 28 17:41:47.202357 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.202257    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p" event={"ID":"b42ae312-9026-4b68-8f00-619b799247d5","Type":"ContainerDied","Data":"374401b2ac6f94211d26a7069fe3e9fc6092a12e92dbda50792818266f2fc90f"}
Jul 28 17:41:47.202357 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.202281    2560 scope.go:117] "RemoveContainer" containerID="2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947"
Jul 28 17:41:47.203594 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.203571    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3cc09d7-9bcd-4493-a44f-d58b48e8d56b" containerID="a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20" exitCode=0
Jul 28 17:41:47.203689 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.203610    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"
Jul 28 17:41:47.203689 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.203675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d" event={"ID":"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b","Type":"ContainerDied","Data":"a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20"}
Jul 28 17:41:47.203806 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.203729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d" event={"ID":"e3cc09d7-9bcd-4493-a44f-d58b48e8d56b","Type":"ContainerDied","Data":"3b57582cb7d46c893688aaffcf7da11738fc44e0645709200d67a0cf79cb806b"}
Jul 28 17:41:47.205112 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.205089    2560 generic.go:356] "Generic (PLEG): container finished" podID="1d243d0f-9903-4b0a-a56f-01714bfb29be" containerID="e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2" exitCode=0
Jul 28 17:41:47.205228 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.205132    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"
Jul 28 17:41:47.205228 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.205139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7" event={"ID":"1d243d0f-9903-4b0a-a56f-01714bfb29be","Type":"ContainerDied","Data":"e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2"}
Jul 28 17:41:47.205228 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.205196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7" event={"ID":"1d243d0f-9903-4b0a-a56f-01714bfb29be","Type":"ContainerDied","Data":"58c9229996c070ca65afbbde45591dc211c0f42159ec003a89aecbb288145804"}
Jul 28 17:41:47.212170 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.212148    2560 scope.go:117] "RemoveContainer" containerID="2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947"
Jul 28 17:41:47.212451 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:41:47.212429    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947\": container with ID starting with 2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947 not found: ID does not exist" containerID="2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947"
Jul 28 17:41:47.212559 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.212459    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947"} err="failed to get container status \"2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947\": rpc error: code = NotFound desc = could not find container \"2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947\": container with ID starting with 2f990ccd67612ce91d9383665fac27e50db058d20f4e2cced9f5a10f57516947 not found: ID does not exist"
Jul 28 17:41:47.212559 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.212483    2560 scope.go:117] "RemoveContainer" containerID="a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20"
Jul 28 17:41:47.220274 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.220257    2560 scope.go:117] "RemoveContainer" containerID="a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20"
Jul 28 17:41:47.220556 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:41:47.220535    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20\": container with ID starting with a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20 not found: ID does not exist" containerID="a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20"
Jul 28 17:41:47.220610 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.220565    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20"} err="failed to get container status \"a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20\": rpc error: code = NotFound desc = could not find container \"a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20\": container with ID starting with a8c33bbe8051594138c6a977412b1a52faf04a0392bff2c30a24815685202f20 not found: ID does not exist"
Jul 28 17:41:47.220610 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.220587    2560 scope.go:117] "RemoveContainer" containerID="e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2"
Jul 28 17:41:47.229137 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.229107    2560 scope.go:117] "RemoveContainer" containerID="e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2"
Jul 28 17:41:47.229405 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:41:47.229381    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2\": container with ID starting with e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2 not found: ID does not exist" containerID="e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2"
Jul 28 17:41:47.229496 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.229410    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2"} err="failed to get container status \"e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2\": rpc error: code = NotFound desc = could not find container \"e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2\": container with ID starting with e0962977038a140cc49f28dfa4d69015f7a226e5bd0f97485e63909758e9a9c2 not found: ID does not exist"
Jul 28 17:41:47.270196 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.270172    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"]
Jul 28 17:41:47.275071 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.275045    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42ae312-9026-4b68-8f00-619b799247d5-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:41:47.275180 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.275072    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:41:47.275180 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.275089    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25jj\" (UniqueName: \"kubernetes.io/projected/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b-kube-api-access-k25jj\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:41:47.275180 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.275103    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp699\" (UniqueName: \"kubernetes.io/projected/1d243d0f-9903-4b0a-a56f-01714bfb29be-kube-api-access-zp699\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:41:47.275180 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.275117    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ae312-9026-4b68-8f00-619b799247d5-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:41:47.275180 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.275129    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d243d0f-9903-4b0a-a56f-01714bfb29be-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:41:47.275180 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.275143    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8sbm\" (UniqueName: \"kubernetes.io/projected/b42ae312-9026-4b68-8f00-619b799247d5-kube-api-access-t8sbm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:41:47.277736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.277715    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4794d51a-6fb8d4d7cd-8xw7d"]
Jul 28 17:41:47.307695 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.307673    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"]
Jul 28 17:41:47.316242 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.316217    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4794d51a-7c687bb556-9bdc7"]
Jul 28 17:41:47.340813 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.340789    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"]
Jul 28 17:41:47.351938 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.351917    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4794d51a-74df677776-qlq4p"]
Jul 28 17:41:47.874423 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.874389    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d243d0f-9903-4b0a-a56f-01714bfb29be" path="/var/lib/kubelet/pods/1d243d0f-9903-4b0a-a56f-01714bfb29be/volumes"
Jul 28 17:41:47.874825 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.874777    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b42ae312-9026-4b68-8f00-619b799247d5" path="/var/lib/kubelet/pods/b42ae312-9026-4b68-8f00-619b799247d5/volumes"
Jul 28 17:41:47.875072 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:47.875058    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3cc09d7-9bcd-4493-a44f-d58b48e8d56b" path="/var/lib/kubelet/pods/e3cc09d7-9bcd-4493-a44f-d58b48e8d56b/volumes"
Jul 28 17:41:59.619805 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:41:59.619769    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"]
Jul 28 17:42:14.638157 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:14.638105    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9" podUID="92a0ff7a-843c-48e1-8b9e-3c57dc772830" containerName="main" containerID="cri-o://ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3" gracePeriod=45
Jul 28 17:42:14.883277 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:14.883249    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:42:15.011613 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011526    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbk2j\" (UniqueName: \"kubernetes.io/projected/92a0ff7a-843c-48e1-8b9e-3c57dc772830-kube-api-access-xbk2j\") pod \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") "
Jul 28 17:42:15.011613 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011572    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tmp-dir\") pod \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") "
Jul 28 17:42:15.011613 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011608    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-model-cache\") pod \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") "
Jul 28 17:42:15.011883 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011651    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-home\") pod \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") "
Jul 28 17:42:15.011883 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011668    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tls-certs\") pod \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") "
Jul 28 17:42:15.011883 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011716    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-dshm\") pod \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\" (UID: \"92a0ff7a-843c-48e1-8b9e-3c57dc772830\") "
Jul 28 17:42:15.012057 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-model-cache" (OuterVolumeSpecName: "model-cache") pod "92a0ff7a-843c-48e1-8b9e-3c57dc772830" (UID: "92a0ff7a-843c-48e1-8b9e-3c57dc772830"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:15.012057 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-home" (OuterVolumeSpecName: "home") pod "92a0ff7a-843c-48e1-8b9e-3c57dc772830" (UID: "92a0ff7a-843c-48e1-8b9e-3c57dc772830"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:15.012057 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.011915    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92a0ff7a-843c-48e1-8b9e-3c57dc772830" (UID: "92a0ff7a-843c-48e1-8b9e-3c57dc772830"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:15.013807 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.013783    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a0ff7a-843c-48e1-8b9e-3c57dc772830-kube-api-access-xbk2j" (OuterVolumeSpecName: "kube-api-access-xbk2j") pod "92a0ff7a-843c-48e1-8b9e-3c57dc772830" (UID: "92a0ff7a-843c-48e1-8b9e-3c57dc772830"). InnerVolumeSpecName "kube-api-access-xbk2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:15.013902 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.013827    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-dshm" (OuterVolumeSpecName: "dshm") pod "92a0ff7a-843c-48e1-8b9e-3c57dc772830" (UID: "92a0ff7a-843c-48e1-8b9e-3c57dc772830"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:15.013902 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.013828    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92a0ff7a-843c-48e1-8b9e-3c57dc772830" (UID: "92a0ff7a-843c-48e1-8b9e-3c57dc772830"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:15.112376 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.112330    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:42:15.112376 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.112372    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:42:15.112376 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.112380    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:42:15.112537 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.112388    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:42:15.112537 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.112397    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbk2j\" (UniqueName: \"kubernetes.io/projected/92a0ff7a-843c-48e1-8b9e-3c57dc772830-kube-api-access-xbk2j\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:42:15.112537 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.112407    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a0ff7a-843c-48e1-8b9e-3c57dc772830-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:42:15.334463 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.334430    2560 generic.go:356] "Generic (PLEG): container finished" podID="92a0ff7a-843c-48e1-8b9e-3c57dc772830" containerID="ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3" exitCode=0
Jul 28 17:42:15.334615 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.334477    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"
Jul 28 17:42:15.334615 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.334515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9" event={"ID":"92a0ff7a-843c-48e1-8b9e-3c57dc772830","Type":"ContainerDied","Data":"ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3"}
Jul 28 17:42:15.334615 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.334559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9" event={"ID":"92a0ff7a-843c-48e1-8b9e-3c57dc772830","Type":"ContainerDied","Data":"6c138c2fb0bbd6e2158857b7a1d8a6763994b185c1b2dfdf9e82ee6ee56f03fb"}
Jul 28 17:42:15.334615 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.334578    2560 scope.go:117] "RemoveContainer" containerID="ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3"
Jul 28 17:42:15.344004 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.343989    2560 scope.go:117] "RemoveContainer" containerID="ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3"
Jul 28 17:42:15.344277 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:42:15.344257    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3\": container with ID starting with ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3 not found: ID does not exist" containerID="ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3"
Jul 28 17:42:15.344325 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.344284    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3"} err="failed to get container status \"ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3\": rpc error: code = NotFound desc = could not find container \"ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3\": container with ID starting with ea1bd258ca51e3bafcc39a929756a190b0cc22a02fd9fb14a01d17c13ce9e9c3 not found: ID does not exist"
Jul 28 17:42:15.359666 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.359645    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"]
Jul 28 17:42:15.363413 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.363392    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4794d51a/e2e-lifecycle-model-4794d51a-kserve-77c6df597f-2dqs9"]
Jul 28 17:42:15.872640 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:42:15.872609    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92a0ff7a-843c-48e1-8b9e-3c57dc772830" path="/var/lib/kubelet/pods/92a0ff7a-843c-48e1-8b9e-3c57dc772830/volumes"
Jul 28 17:43:45.904967 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.904935    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"]
Jul 28 17:43:45.905369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905312    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cc09d7-9bcd-4493-a44f-d58b48e8d56b" containerName="payload-processing"
Jul 28 17:43:45.905369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905324    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cc09d7-9bcd-4493-a44f-d58b48e8d56b" containerName="payload-processing"
Jul 28 17:43:45.905369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905334    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a0ff7a-843c-48e1-8b9e-3c57dc772830" containerName="main"
Jul 28 17:43:45.905369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905361    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a0ff7a-843c-48e1-8b9e-3c57dc772830" containerName="main"
Jul 28 17:43:45.905505 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905376    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42ae312-9026-4b68-8f00-619b799247d5" containerName="maas-api"
Jul 28 17:43:45.905505 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905383    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42ae312-9026-4b68-8f00-619b799247d5" containerName="maas-api"
Jul 28 17:43:45.905505 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905389    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d243d0f-9903-4b0a-a56f-01714bfb29be" containerName="payload-pre-processing"
Jul 28 17:43:45.905505 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905394    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d243d0f-9903-4b0a-a56f-01714bfb29be" containerName="payload-pre-processing"
Jul 28 17:43:45.905505 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905468    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42ae312-9026-4b68-8f00-619b799247d5" containerName="maas-api"
Jul 28 17:43:45.905505 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905476    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cc09d7-9bcd-4493-a44f-d58b48e8d56b" containerName="payload-processing"
Jul 28 17:43:45.905505 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905485    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d243d0f-9903-4b0a-a56f-01714bfb29be" containerName="payload-pre-processing"
Jul 28 17:43:45.905505 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.905491    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a0ff7a-843c-48e1-8b9e-3c57dc772830" containerName="main"
Jul 28 17:43:45.908720 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.908701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:45.911262 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.911238    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:43:45.911389 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.911299    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7dbdbf-serving-cert\""
Jul 28 17:43:45.911389 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.911313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:43:45.911589 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.911573    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:43:45.919523 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:45.919502    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"]
Jul 28 17:43:46.103735 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.103701    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdsn\" (UniqueName: \"kubernetes.io/projected/431f0b54-569d-49c8-936c-e8c400f47a88-kube-api-access-rzdsn\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.103910 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.103749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.103910 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.103786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431f0b54-569d-49c8-936c-e8c400f47a88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.204399 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.204307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdsn\" (UniqueName: \"kubernetes.io/projected/431f0b54-569d-49c8-936c-e8c400f47a88-kube-api-access-rzdsn\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.204399 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.204375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.204593 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.204404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431f0b54-569d-49c8-936c-e8c400f47a88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.204593 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:43:46.204512    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-serving-cert: secret "maas-api-e2e-inf-a-7dbdbf-serving-cert" not found
Jul 28 17:43:46.204593 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:43:46.204581    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls podName:431f0b54-569d-49c8-936c-e8c400f47a88 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:46.704564344 +0000 UTC m=+2097.376892247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls") pod "maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm" (UID: "431f0b54-569d-49c8-936c-e8c400f47a88") : secret "maas-api-e2e-inf-a-7dbdbf-serving-cert" not found
Jul 28 17:43:46.205020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.205003    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431f0b54-569d-49c8-936c-e8c400f47a88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.214985 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.214957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdsn\" (UniqueName: \"kubernetes.io/projected/431f0b54-569d-49c8-936c-e8c400f47a88-kube-api-access-rzdsn\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.605273 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.605240    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"]
Jul 28 17:43:46.609120 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.609099    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:46.609600 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.609574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhbj9\" (UniqueName: \"kubernetes.io/projected/a34d8187-0213-4671-b7da-9e04bc92f00a-kube-api-access-qhbj9\") pod \"payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv\" (UID: \"a34d8187-0213-4671-b7da-9e04bc92f00a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:46.609737 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.609719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34d8187-0213-4671-b7da-9e04bc92f00a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv\" (UID: \"a34d8187-0213-4671-b7da-9e04bc92f00a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:46.611713 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.611689    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7dbdbf-dockercfg-5ftws\""
Jul 28 17:43:46.611799 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.611722    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7dbdbf\""
Jul 28 17:43:46.619513 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.619490    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"]
Jul 28 17:43:46.641973 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.641948    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"]
Jul 28 17:43:46.645402 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.645384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:46.657624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.657602    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"]
Jul 28 17:43:46.710251 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.710219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhbj9\" (UniqueName: \"kubernetes.io/projected/a34d8187-0213-4671-b7da-9e04bc92f00a-kube-api-access-qhbj9\") pod \"payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv\" (UID: \"a34d8187-0213-4671-b7da-9e04bc92f00a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:46.710395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.710263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdhr\" (UniqueName: \"kubernetes.io/projected/51b9d714-962e-4fc4-b251-1d9151aa3488-kube-api-access-hsdhr\") pod \"payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt\" (UID: \"51b9d714-962e-4fc4-b251-1d9151aa3488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:46.710395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.710309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.710395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.710332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34d8187-0213-4671-b7da-9e04bc92f00a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv\" (UID: \"a34d8187-0213-4671-b7da-9e04bc92f00a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:46.710395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.710360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b9d714-962e-4fc4-b251-1d9151aa3488-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt\" (UID: \"51b9d714-962e-4fc4-b251-1d9151aa3488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:46.711013 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.710993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34d8187-0213-4671-b7da-9e04bc92f00a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv\" (UID: \"a34d8187-0213-4671-b7da-9e04bc92f00a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:46.712590 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.712570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.719215 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.719196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhbj9\" (UniqueName: \"kubernetes.io/projected/a34d8187-0213-4671-b7da-9e04bc92f00a-kube-api-access-qhbj9\") pod \"payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv\" (UID: \"a34d8187-0213-4671-b7da-9e04bc92f00a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:46.810913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.810885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdhr\" (UniqueName: \"kubernetes.io/projected/51b9d714-962e-4fc4-b251-1d9151aa3488-kube-api-access-hsdhr\") pod \"payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt\" (UID: \"51b9d714-962e-4fc4-b251-1d9151aa3488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:46.811034 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.810945    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b9d714-962e-4fc4-b251-1d9151aa3488-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt\" (UID: \"51b9d714-962e-4fc4-b251-1d9151aa3488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:46.811608 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.811591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b9d714-962e-4fc4-b251-1d9151aa3488-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt\" (UID: \"51b9d714-962e-4fc4-b251-1d9151aa3488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:46.819601 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.819577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdhr\" (UniqueName: \"kubernetes.io/projected/51b9d714-962e-4fc4-b251-1d9151aa3488-kube-api-access-hsdhr\") pod \"payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt\" (UID: \"51b9d714-962e-4fc4-b251-1d9151aa3488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:46.820365 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.820335    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:46.919759 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.919725    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:46.951170 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.951145    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"]
Jul 28 17:43:46.953057 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:43:46.953032    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431f0b54_569d_49c8_936c_e8c400f47a88.slice/crio-54a02dda3fe25ee5abc117ffa56939f9e3660b13823685f34937a362571c4dbd WatchSource:0}: Error finding container 54a02dda3fe25ee5abc117ffa56939f9e3660b13823685f34937a362571c4dbd: Status 404 returned error can't find the container with id 54a02dda3fe25ee5abc117ffa56939f9e3660b13823685f34937a362571c4dbd
Jul 28 17:43:46.954555 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:46.954536    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:47.064029 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.063998    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"]
Jul 28 17:43:47.067753 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:43:47.067713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34d8187_0213_4671_b7da_9e04bc92f00a.slice/crio-ea0194236c862802ce38fdb5a5d816e703480a6f661e071287cdf7d2326f8304 WatchSource:0}: Error finding container ea0194236c862802ce38fdb5a5d816e703480a6f661e071287cdf7d2326f8304: Status 404 returned error can't find the container with id ea0194236c862802ce38fdb5a5d816e703480a6f661e071287cdf7d2326f8304
Jul 28 17:43:47.099565 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.099543    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"]
Jul 28 17:43:47.103532 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:43:47.103506    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b9d714_962e_4fc4_b251_1d9151aa3488.slice/crio-90197a265b76f0e902bbbd36e8685cb84cb486d71a26df595fbdcefd5f4eba10 WatchSource:0}: Error finding container 90197a265b76f0e902bbbd36e8685cb84cb486d71a26df595fbdcefd5f4eba10: Status 404 returned error can't find the container with id 90197a265b76f0e902bbbd36e8685cb84cb486d71a26df595fbdcefd5f4eba10
Jul 28 17:43:47.758413 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.758301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv" event={"ID":"a34d8187-0213-4671-b7da-9e04bc92f00a","Type":"ContainerStarted","Data":"737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24"}
Jul 28 17:43:47.758413 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.758388    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv" event={"ID":"a34d8187-0213-4671-b7da-9e04bc92f00a","Type":"ContainerStarted","Data":"ea0194236c862802ce38fdb5a5d816e703480a6f661e071287cdf7d2326f8304"}
Jul 28 17:43:47.758413 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.758413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:47.760056 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.760024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt" event={"ID":"51b9d714-962e-4fc4-b251-1d9151aa3488","Type":"ContainerStarted","Data":"f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438"}
Jul 28 17:43:47.760173 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.760063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt" event={"ID":"51b9d714-962e-4fc4-b251-1d9151aa3488","Type":"ContainerStarted","Data":"90197a265b76f0e902bbbd36e8685cb84cb486d71a26df595fbdcefd5f4eba10"}
Jul 28 17:43:47.760173 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.760147    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:43:47.761537 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.761514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm" event={"ID":"431f0b54-569d-49c8-936c-e8c400f47a88","Type":"ContainerStarted","Data":"08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e"}
Jul 28 17:43:47.761637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.761558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm" event={"ID":"431f0b54-569d-49c8-936c-e8c400f47a88","Type":"ContainerStarted","Data":"54a02dda3fe25ee5abc117ffa56939f9e3660b13823685f34937a362571c4dbd"}
Jul 28 17:43:47.761717 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.761646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:47.785681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.785639    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv" podStartSLOduration=1.785626437 podStartE2EDuration="1.785626437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:47.78342507 +0000 UTC m=+2098.455752996" watchObservedRunningTime="2026-07-28 17:43:47.785626437 +0000 UTC m=+2098.457954420"
Jul 28 17:43:47.806172 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.806132    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt" podStartSLOduration=1.806118793 podStartE2EDuration="1.806118793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:47.804590285 +0000 UTC m=+2098.476918201" watchObservedRunningTime="2026-07-28 17:43:47.806118793 +0000 UTC m=+2098.478446751"
Jul 28 17:43:47.826646 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:47.826612    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm" podStartSLOduration=2.361327835 podStartE2EDuration="2.826598966s" totalImagesPullingTime="465.271131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:47.824741428 +0000 UTC m=+2098.497069372" watchObservedRunningTime="2026-07-28 17:43:47.826598966 +0000 UTC m=+2098.498926864"
Jul 28 17:43:49.906496 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:49.906468    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:43:49.912122 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:49.912101    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:43:53.771723 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:53.771693    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:43:54.109782 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.109753    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"]
Jul 28 17:43:54.113155 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.113137    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.115771 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.115749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8796a3-serving-cert\""
Jul 28 17:43:54.125999 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.125974    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"]
Jul 28 17:43:54.163233 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.163210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zkjf\" (UniqueName: \"kubernetes.io/projected/253789e1-f5d7-4881-9cb8-32c352fa18fe-kube-api-access-6zkjf\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.163333 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.163239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.163333 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.163289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253789e1-f5d7-4881-9cb8-32c352fa18fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.263898 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.263872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253789e1-f5d7-4881-9cb8-32c352fa18fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.264020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.264008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkjf\" (UniqueName: \"kubernetes.io/projected/253789e1-f5d7-4881-9cb8-32c352fa18fe-kube-api-access-6zkjf\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.264071 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.264029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.264141 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:43:54.264128    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-serving-cert: secret "maas-api-e2e-inf-b-8796a3-serving-cert" not found
Jul 28 17:43:54.264192 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:43:54.264180    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls podName:253789e1-f5d7-4881-9cb8-32c352fa18fe nodeName:}" failed. No retries permitted until 2026-07-28 17:43:54.764162216 +0000 UTC m=+2105.436490135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls") pod "maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4" (UID: "253789e1-f5d7-4881-9cb8-32c352fa18fe") : secret "maas-api-e2e-inf-b-8796a3-serving-cert" not found
Jul 28 17:43:54.264541 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.264517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253789e1-f5d7-4881-9cb8-32c352fa18fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.276633 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.276614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkjf\" (UniqueName: \"kubernetes.io/projected/253789e1-f5d7-4881-9cb8-32c352fa18fe-kube-api-access-6zkjf\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.767101 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.767070    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.769364 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.769331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:54.796817 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.796786    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"]
Jul 28 17:43:54.801193 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.801174    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:54.803544 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.803521    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8796a3\""
Jul 28 17:43:54.803673 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.803562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8796a3-dockercfg-p2h47\""
Jul 28 17:43:54.812034 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.812011    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"]
Jul 28 17:43:54.846708 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.846682    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"]
Jul 28 17:43:54.849956 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.849941    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:54.859905 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.859885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"]
Jul 28 17:43:54.868181 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.868160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczpf\" (UniqueName: \"kubernetes.io/projected/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-kube-api-access-sczpf\") pod \"payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf\" (UID: \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:54.868271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.868220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2\" (UID: \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:54.868271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.868246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr2fs\" (UniqueName: \"kubernetes.io/projected/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-kube-api-access-sr2fs\") pod \"payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2\" (UID: \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:54.868271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.868261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf\" (UID: \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:54.969074 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.969043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczpf\" (UniqueName: \"kubernetes.io/projected/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-kube-api-access-sczpf\") pod \"payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf\" (UID: \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:54.969244 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.969189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2\" (UID: \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:54.969244 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.969219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2fs\" (UniqueName: \"kubernetes.io/projected/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-kube-api-access-sr2fs\") pod \"payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2\" (UID: \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:54.969244 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.969239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf\" (UID: \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:54.969962 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.969940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf\" (UID: \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:54.970049 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.969943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2\" (UID: \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:54.978023 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.978002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr2fs\" (UniqueName: \"kubernetes.io/projected/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-kube-api-access-sr2fs\") pod \"payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2\" (UID: \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:54.978105 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:54.978037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczpf\" (UniqueName: \"kubernetes.io/projected/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-kube-api-access-sczpf\") pod \"payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf\" (UID: \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:55.023130 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.023062    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:55.114065 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.114029    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:55.156920 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.155819    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"]
Jul 28 17:43:55.158601 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:43:55.158567    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253789e1_f5d7_4881_9cb8_32c352fa18fe.slice/crio-c51b7f6082ce9fe802adae485b69c6c916e8be75c3dbd7ed837fc22c27f7281d WatchSource:0}: Error finding container c51b7f6082ce9fe802adae485b69c6c916e8be75c3dbd7ed837fc22c27f7281d: Status 404 returned error can't find the container with id c51b7f6082ce9fe802adae485b69c6c916e8be75c3dbd7ed837fc22c27f7281d
Jul 28 17:43:55.159681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.159660    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:55.260517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.260490    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"]
Jul 28 17:43:55.263750 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:43:55.263715    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb1ba0b_f6e1_4de8_a482_75cc0220edac.slice/crio-fc3cacbf4143614525a5ec8b20e51168f9a8f99eb02706e77ccd7c3881fbbdef WatchSource:0}: Error finding container fc3cacbf4143614525a5ec8b20e51168f9a8f99eb02706e77ccd7c3881fbbdef: Status 404 returned error can't find the container with id fc3cacbf4143614525a5ec8b20e51168f9a8f99eb02706e77ccd7c3881fbbdef
Jul 28 17:43:55.307291 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.307242    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"]
Jul 28 17:43:55.312429 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:43:55.312403    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5bed1d_0ac1_4a0c_8631_f036cac5ea84.slice/crio-a895d0e9cb1f971cb4cb58159a703239ae1eff02450446eed83d11a9ee7d06c4 WatchSource:0}: Error finding container a895d0e9cb1f971cb4cb58159a703239ae1eff02450446eed83d11a9ee7d06c4: Status 404 returned error can't find the container with id a895d0e9cb1f971cb4cb58159a703239ae1eff02450446eed83d11a9ee7d06c4
Jul 28 17:43:55.800251 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.800225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2" event={"ID":"ecb1ba0b-f6e1-4de8-a482-75cc0220edac","Type":"ContainerStarted","Data":"ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b"}
Jul 28 17:43:55.800645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.800283    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2" event={"ID":"ecb1ba0b-f6e1-4de8-a482-75cc0220edac","Type":"ContainerStarted","Data":"fc3cacbf4143614525a5ec8b20e51168f9a8f99eb02706e77ccd7c3881fbbdef"}
Jul 28 17:43:55.800645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.800316    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:43:55.801659 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.801635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4" event={"ID":"253789e1-f5d7-4881-9cb8-32c352fa18fe","Type":"ContainerStarted","Data":"c51b7f6082ce9fe802adae485b69c6c916e8be75c3dbd7ed837fc22c27f7281d"}
Jul 28 17:43:55.803089 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.803070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf" event={"ID":"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84","Type":"ContainerStarted","Data":"ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a"}
Jul 28 17:43:55.803202 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.803115    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:43:55.803202 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.803140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf" event={"ID":"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84","Type":"ContainerStarted","Data":"a895d0e9cb1f971cb4cb58159a703239ae1eff02450446eed83d11a9ee7d06c4"}
Jul 28 17:43:55.822783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.822744    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2" podStartSLOduration=1.822730005 podStartE2EDuration="1.822730005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:55.820600092 +0000 UTC m=+2106.492928015" watchObservedRunningTime="2026-07-28 17:43:55.822730005 +0000 UTC m=+2106.495057926"
Jul 28 17:43:55.838562 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:55.838334    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf" podStartSLOduration=1.838318122 podStartE2EDuration="1.838318122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:55.837760617 +0000 UTC m=+2106.510088536" watchObservedRunningTime="2026-07-28 17:43:55.838318122 +0000 UTC m=+2106.510646040"
Jul 28 17:43:56.809517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:56.809467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4" event={"ID":"253789e1-f5d7-4881-9cb8-32c352fa18fe","Type":"ContainerStarted","Data":"3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0"}
Jul 28 17:43:56.809973 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:56.809858    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:43:56.829378 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:56.829316    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4" podStartSLOduration=2.1934929260000002 podStartE2EDuration="2.829296693s" totalImagesPullingTime="635.803767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:56.82788632 +0000 UTC m=+2107.500214240" watchObservedRunningTime="2026-07-28 17:43:56.829296693 +0000 UTC m=+2107.501624613"
Jul 28 17:43:58.769445 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:58.769413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:43:58.769898 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:43:58.769573    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:44:03.831757 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:03.831729    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:44:06.811371 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:06.811318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:44:06.811913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:06.811418    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:44:12.107032 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.106996    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"]
Jul 28 17:44:12.110449 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.110421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.114003 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.113978    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7dbdbf\"/\"openshift-service-ca.crt\""
Jul 28 17:44:12.114133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.114031    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7dbdbf\"/\"test-model-7dbdbf-kserve-self-signed-certs\""
Jul 28 17:44:12.114133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.114044    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7dbdbf\"/\"default-dockercfg-txzgt\""
Jul 28 17:44:12.114133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.114061    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7dbdbf\"/\"kube-root-ca.crt\""
Jul 28 17:44:12.123501 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.123479    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"]
Jul 28 17:44:12.210226 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.210197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-tmp-dir\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.210380 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.210263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-home\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.210380 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.210290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-model-cache\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.210380 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.210315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5714fb99-901d-41ae-a54d-f76d6c0c097a-tls-certs\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.210380 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.210337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-dshm\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.210563 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.210413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klf59\" (UniqueName: \"kubernetes.io/projected/5714fb99-901d-41ae-a54d-f76d6c0c097a-kube-api-access-klf59\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311038 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.310998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-home\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311222 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.311046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-model-cache\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311299 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.311218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5714fb99-901d-41ae-a54d-f76d6c0c097a-tls-certs\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311299 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.311265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-dshm\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311446 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.311431    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-model-cache\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311508 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.311437    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klf59\" (UniqueName: \"kubernetes.io/projected/5714fb99-901d-41ae-a54d-f76d6c0c097a-kube-api-access-klf59\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311508 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.311491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-home\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311592 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.311528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-tmp-dir\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.311825 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.311799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-tmp-dir\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.313386 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.313365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-dshm\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.313614 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.313598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5714fb99-901d-41ae-a54d-f76d6c0c097a-tls-certs\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.322131 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.322107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klf59\" (UniqueName: \"kubernetes.io/projected/5714fb99-901d-41ae-a54d-f76d6c0c097a-kube-api-access-klf59\") pod \"test-model-7dbdbf-kserve-58c578948c-7hvn4\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") " pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.419975 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.419912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:12.550699 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.550674    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"]
Jul 28 17:44:12.553194 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:44:12.553159    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5714fb99_901d_41ae_a54d_f76d6c0c097a.slice/crio-093f86c9c338b3f0496e9e2402497a4c4a0a1064fb812512d35a0cdcdec99170 WatchSource:0}: Error finding container 093f86c9c338b3f0496e9e2402497a4c4a0a1064fb812512d35a0cdcdec99170: Status 404 returned error can't find the container with id 093f86c9c338b3f0496e9e2402497a4c4a0a1064fb812512d35a0cdcdec99170
Jul 28 17:44:12.902812 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.902781    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4" event={"ID":"5714fb99-901d-41ae-a54d-f76d6c0c097a","Type":"ContainerStarted","Data":"23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046"}
Jul 28 17:44:12.902976 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.902817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4" event={"ID":"5714fb99-901d-41ae-a54d-f76d6c0c097a","Type":"ContainerStarted","Data":"093f86c9c338b3f0496e9e2402497a4c4a0a1064fb812512d35a0cdcdec99170"}
Jul 28 17:44:12.923621 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:12.923576    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4" podStartSLOduration=0.923560126 podStartE2EDuration="923.560126ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:12.921363367 +0000 UTC m=+2123.593691282" watchObservedRunningTime="2026-07-28 17:44:12.923560126 +0000 UTC m=+2123.595888045"
Jul 28 17:44:22.420703 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:22.420668    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:22.421074 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:22.420746    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:22.432964 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:22.432938    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:44:22.959631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:44:22.959599    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:45:07.886575 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:07.886543    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"]
Jul 28 17:45:19.463411 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.463371    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"]
Jul 28 17:45:19.464003 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.463643    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm" podUID="431f0b54-569d-49c8-936c-e8c400f47a88" containerName="maas-api" containerID="cri-o://08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e" gracePeriod=30
Jul 28 17:45:19.530309 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.530271    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"]
Jul 28 17:45:19.530548 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.530517    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv" podUID="a34d8187-0213-4671-b7da-9e04bc92f00a" containerName="payload-processing" containerID="cri-o://737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24" gracePeriod=30
Jul 28 17:45:19.539055 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.539024    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"]
Jul 28 17:45:19.539698 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.539616    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt" podUID="51b9d714-962e-4fc4-b251-1d9151aa3488" containerName="payload-pre-processing" containerID="cri-o://f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438" gracePeriod=30
Jul 28 17:45:19.820174 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.820148    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:45:19.843694 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.843666    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:45:19.855559 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.855530    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhbj9\" (UniqueName: \"kubernetes.io/projected/a34d8187-0213-4671-b7da-9e04bc92f00a-kube-api-access-qhbj9\") pod \"a34d8187-0213-4671-b7da-9e04bc92f00a\" (UID: \"a34d8187-0213-4671-b7da-9e04bc92f00a\") "
Jul 28 17:45:19.855700 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.855584    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsdhr\" (UniqueName: \"kubernetes.io/projected/51b9d714-962e-4fc4-b251-1d9151aa3488-kube-api-access-hsdhr\") pod \"51b9d714-962e-4fc4-b251-1d9151aa3488\" (UID: \"51b9d714-962e-4fc4-b251-1d9151aa3488\") "
Jul 28 17:45:19.855700 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.855616    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b9d714-962e-4fc4-b251-1d9151aa3488-plugins-config-volume\") pod \"51b9d714-962e-4fc4-b251-1d9151aa3488\" (UID: \"51b9d714-962e-4fc4-b251-1d9151aa3488\") "
Jul 28 17:45:19.855700 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.855644    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34d8187-0213-4671-b7da-9e04bc92f00a-plugins-config-volume\") pod \"a34d8187-0213-4671-b7da-9e04bc92f00a\" (UID: \"a34d8187-0213-4671-b7da-9e04bc92f00a\") "
Jul 28 17:45:19.856065 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.856036    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b9d714-962e-4fc4-b251-1d9151aa3488-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51b9d714-962e-4fc4-b251-1d9151aa3488" (UID: "51b9d714-962e-4fc4-b251-1d9151aa3488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:19.856185 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.856111    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34d8187-0213-4671-b7da-9e04bc92f00a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a34d8187-0213-4671-b7da-9e04bc92f00a" (UID: "a34d8187-0213-4671-b7da-9e04bc92f00a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:19.858165 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.858142    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34d8187-0213-4671-b7da-9e04bc92f00a-kube-api-access-qhbj9" (OuterVolumeSpecName: "kube-api-access-qhbj9") pod "a34d8187-0213-4671-b7da-9e04bc92f00a" (UID: "a34d8187-0213-4671-b7da-9e04bc92f00a"). InnerVolumeSpecName "kube-api-access-qhbj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:19.858165 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.858150    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b9d714-962e-4fc4-b251-1d9151aa3488-kube-api-access-hsdhr" (OuterVolumeSpecName: "kube-api-access-hsdhr") pod "51b9d714-962e-4fc4-b251-1d9151aa3488" (UID: "51b9d714-962e-4fc4-b251-1d9151aa3488"). InnerVolumeSpecName "kube-api-access-hsdhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:19.904459 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.904433    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:45:19.957163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.957135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzdsn\" (UniqueName: \"kubernetes.io/projected/431f0b54-569d-49c8-936c-e8c400f47a88-kube-api-access-rzdsn\") pod \"431f0b54-569d-49c8-936c-e8c400f47a88\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") "
Jul 28 17:45:19.957322 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.957227    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431f0b54-569d-49c8-936c-e8c400f47a88-openshift-service-ca\") pod \"431f0b54-569d-49c8-936c-e8c400f47a88\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") "
Jul 28 17:45:19.957322 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.957275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls\") pod \"431f0b54-569d-49c8-936c-e8c400f47a88\" (UID: \"431f0b54-569d-49c8-936c-e8c400f47a88\") "
Jul 28 17:45:19.957478 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.957466    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b9d714-962e-4fc4-b251-1d9151aa3488-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:19.957530 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.957480    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34d8187-0213-4671-b7da-9e04bc92f00a-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:19.957530 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.957489    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhbj9\" (UniqueName: \"kubernetes.io/projected/a34d8187-0213-4671-b7da-9e04bc92f00a-kube-api-access-qhbj9\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:19.957530 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.957497    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsdhr\" (UniqueName: \"kubernetes.io/projected/51b9d714-962e-4fc4-b251-1d9151aa3488-kube-api-access-hsdhr\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:19.957627 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.957587    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431f0b54-569d-49c8-936c-e8c400f47a88-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "431f0b54-569d-49c8-936c-e8c400f47a88" (UID: "431f0b54-569d-49c8-936c-e8c400f47a88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:19.959271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.959250    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431f0b54-569d-49c8-936c-e8c400f47a88-kube-api-access-rzdsn" (OuterVolumeSpecName: "kube-api-access-rzdsn") pod "431f0b54-569d-49c8-936c-e8c400f47a88" (UID: "431f0b54-569d-49c8-936c-e8c400f47a88"). InnerVolumeSpecName "kube-api-access-rzdsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:19.959331 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:19.959274    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "431f0b54-569d-49c8-936c-e8c400f47a88" (UID: "431f0b54-569d-49c8-936c-e8c400f47a88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:20.058718 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.058685    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431f0b54-569d-49c8-936c-e8c400f47a88-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:20.058718 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.058713    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431f0b54-569d-49c8-936c-e8c400f47a88-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:20.058718 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.058722    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdsn\" (UniqueName: \"kubernetes.io/projected/431f0b54-569d-49c8-936c-e8c400f47a88-kube-api-access-rzdsn\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:20.225847 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.225813    2560 generic.go:356] "Generic (PLEG): container finished" podID="a34d8187-0213-4671-b7da-9e04bc92f00a" containerID="737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24" exitCode=0
Jul 28 17:45:20.226002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.225855    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"
Jul 28 17:45:20.226002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.225888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv" event={"ID":"a34d8187-0213-4671-b7da-9e04bc92f00a","Type":"ContainerDied","Data":"737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24"}
Jul 28 17:45:20.226002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.225929    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv" event={"ID":"a34d8187-0213-4671-b7da-9e04bc92f00a","Type":"ContainerDied","Data":"ea0194236c862802ce38fdb5a5d816e703480a6f661e071287cdf7d2326f8304"}
Jul 28 17:45:20.226002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.225956    2560 scope.go:117] "RemoveContainer" containerID="737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24"
Jul 28 17:45:20.227465 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.227438    2560 generic.go:356] "Generic (PLEG): container finished" podID="51b9d714-962e-4fc4-b251-1d9151aa3488" containerID="f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438" exitCode=0
Jul 28 17:45:20.227618 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.227475    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"
Jul 28 17:45:20.227618 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.227528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt" event={"ID":"51b9d714-962e-4fc4-b251-1d9151aa3488","Type":"ContainerDied","Data":"f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438"}
Jul 28 17:45:20.227618 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.227570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt" event={"ID":"51b9d714-962e-4fc4-b251-1d9151aa3488","Type":"ContainerDied","Data":"90197a265b76f0e902bbbd36e8685cb84cb486d71a26df595fbdcefd5f4eba10"}
Jul 28 17:45:20.229265 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.229242    2560 generic.go:356] "Generic (PLEG): container finished" podID="431f0b54-569d-49c8-936c-e8c400f47a88" containerID="08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e" exitCode=0
Jul 28 17:45:20.229404 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.229292    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm" event={"ID":"431f0b54-569d-49c8-936c-e8c400f47a88","Type":"ContainerDied","Data":"08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e"}
Jul 28 17:45:20.229404 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.229323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm" event={"ID":"431f0b54-569d-49c8-936c-e8c400f47a88","Type":"ContainerDied","Data":"54a02dda3fe25ee5abc117ffa56939f9e3660b13823685f34937a362571c4dbd"}
Jul 28 17:45:20.229404 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.229293    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"
Jul 28 17:45:20.234635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.234618    2560 scope.go:117] "RemoveContainer" containerID="737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24"
Jul 28 17:45:20.234886 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:45:20.234870    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24\": container with ID starting with 737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24 not found: ID does not exist" containerID="737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24"
Jul 28 17:45:20.234960 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.234892    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24"} err="failed to get container status \"737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24\": rpc error: code = NotFound desc = could not find container \"737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24\": container with ID starting with 737c282c8fbd24763c6a38a3ea2e2878fa175c35bedffc6439c7e8355c711c24 not found: ID does not exist"
Jul 28 17:45:20.234960 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.234907    2560 scope.go:117] "RemoveContainer" containerID="f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438"
Jul 28 17:45:20.242145 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.242132    2560 scope.go:117] "RemoveContainer" containerID="f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438"
Jul 28 17:45:20.242393 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:45:20.242373    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438\": container with ID starting with f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438 not found: ID does not exist" containerID="f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438"
Jul 28 17:45:20.242494 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.242396    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438"} err="failed to get container status \"f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438\": rpc error: code = NotFound desc = could not find container \"f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438\": container with ID starting with f9107bc66e34d0bc492ea6d872cac057c8a0e72584bf12b4510f00cbe5a5c438 not found: ID does not exist"
Jul 28 17:45:20.242494 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.242411    2560 scope.go:117] "RemoveContainer" containerID="08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e"
Jul 28 17:45:20.249546 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.249525    2560 scope.go:117] "RemoveContainer" containerID="08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e"
Jul 28 17:45:20.249834 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:45:20.249789    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e\": container with ID starting with 08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e not found: ID does not exist" containerID="08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e"
Jul 28 17:45:20.249834 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.249813    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e"} err="failed to get container status \"08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e\": rpc error: code = NotFound desc = could not find container \"08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e\": container with ID starting with 08c50e5135a02e737a441452abd548184c7da7d4cc9dafe2b592f50bfed7a76e not found: ID does not exist"
Jul 28 17:45:20.252322 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.252301    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"]
Jul 28 17:45:20.256724 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.256705    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dbdbf-db6cb8496-cn9wt"]
Jul 28 17:45:20.267546 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.267528    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"]
Jul 28 17:45:20.275327 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.275308    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dbdbf-76fbfccf48-mxlmv"]
Jul 28 17:45:20.290133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.290111    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"]
Jul 28 17:45:20.296398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:20.296378    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dbdbf-64c988d466-n68zm"]
Jul 28 17:45:21.872676 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:21.872645    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="431f0b54-569d-49c8-936c-e8c400f47a88" path="/var/lib/kubelet/pods/431f0b54-569d-49c8-936c-e8c400f47a88/volumes"
Jul 28 17:45:21.873071 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:21.873031    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b9d714-962e-4fc4-b251-1d9151aa3488" path="/var/lib/kubelet/pods/51b9d714-962e-4fc4-b251-1d9151aa3488/volumes"
Jul 28 17:45:21.873353 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:21.873326    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34d8187-0213-4671-b7da-9e04bc92f00a" path="/var/lib/kubelet/pods/a34d8187-0213-4671-b7da-9e04bc92f00a/volumes"
Jul 28 17:45:22.905218 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:22.905174    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4" podUID="5714fb99-901d-41ae-a54d-f76d6c0c097a" containerName="main" containerID="cri-o://23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046" gracePeriod=45
Jul 28 17:45:22.949959 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:22.949913    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4" podUID="5714fb99-901d-41ae-a54d-f76d6c0c097a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 28 17:45:23.151504 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.151475    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:45:23.186277 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186208    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klf59\" (UniqueName: \"kubernetes.io/projected/5714fb99-901d-41ae-a54d-f76d6c0c097a-kube-api-access-klf59\") pod \"5714fb99-901d-41ae-a54d-f76d6c0c097a\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") "
Jul 28 17:45:23.186277 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-tmp-dir\") pod \"5714fb99-901d-41ae-a54d-f76d6c0c097a\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") "
Jul 28 17:45:23.186483 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-home\") pod \"5714fb99-901d-41ae-a54d-f76d6c0c097a\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") "
Jul 28 17:45:23.186483 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186407    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5714fb99-901d-41ae-a54d-f76d6c0c097a-tls-certs\") pod \"5714fb99-901d-41ae-a54d-f76d6c0c097a\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") "
Jul 28 17:45:23.186483 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-model-cache\") pod \"5714fb99-901d-41ae-a54d-f76d6c0c097a\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") "
Jul 28 17:45:23.186710 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186513    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-dshm\") pod \"5714fb99-901d-41ae-a54d-f76d6c0c097a\" (UID: \"5714fb99-901d-41ae-a54d-f76d6c0c097a\") "
Jul 28 17:45:23.186710 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186534    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5714fb99-901d-41ae-a54d-f76d6c0c097a" (UID: "5714fb99-901d-41ae-a54d-f76d6c0c097a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:23.186710 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186549    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-home" (OuterVolumeSpecName: "home") pod "5714fb99-901d-41ae-a54d-f76d6c0c097a" (UID: "5714fb99-901d-41ae-a54d-f76d6c0c097a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:23.186710 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186697    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5714fb99-901d-41ae-a54d-f76d6c0c097a" (UID: "5714fb99-901d-41ae-a54d-f76d6c0c097a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:23.186896 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186876    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:23.186932 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186903    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:23.186932 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.186916    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:23.188585 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.188564    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-dshm" (OuterVolumeSpecName: "dshm") pod "5714fb99-901d-41ae-a54d-f76d6c0c097a" (UID: "5714fb99-901d-41ae-a54d-f76d6c0c097a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:23.188903 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.188880    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5714fb99-901d-41ae-a54d-f76d6c0c097a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5714fb99-901d-41ae-a54d-f76d6c0c097a" (UID: "5714fb99-901d-41ae-a54d-f76d6c0c097a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:23.188972 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.188881    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5714fb99-901d-41ae-a54d-f76d6c0c097a-kube-api-access-klf59" (OuterVolumeSpecName: "kube-api-access-klf59") pod "5714fb99-901d-41ae-a54d-f76d6c0c097a" (UID: "5714fb99-901d-41ae-a54d-f76d6c0c097a"). InnerVolumeSpecName "kube-api-access-klf59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:23.251229 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.251195    2560 generic.go:356] "Generic (PLEG): container finished" podID="5714fb99-901d-41ae-a54d-f76d6c0c097a" containerID="23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046" exitCode=0
Jul 28 17:45:23.251407 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.251234    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"
Jul 28 17:45:23.251407 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.251248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4" event={"ID":"5714fb99-901d-41ae-a54d-f76d6c0c097a","Type":"ContainerDied","Data":"23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046"}
Jul 28 17:45:23.251407 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.251285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4" event={"ID":"5714fb99-901d-41ae-a54d-f76d6c0c097a","Type":"ContainerDied","Data":"093f86c9c338b3f0496e9e2402497a4c4a0a1064fb812512d35a0cdcdec99170"}
Jul 28 17:45:23.251407 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.251306    2560 scope.go:117] "RemoveContainer" containerID="23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046"
Jul 28 17:45:23.260132 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.260114    2560 scope.go:117] "RemoveContainer" containerID="23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046"
Jul 28 17:45:23.260393 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:45:23.260375    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046\": container with ID starting with 23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046 not found: ID does not exist" containerID="23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046"
Jul 28 17:45:23.260466 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.260401    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046"} err="failed to get container status \"23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046\": rpc error: code = NotFound desc = could not find container \"23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046\": container with ID starting with 23bb3a425ede852c3e34928b234c04531a101187487863a1f9bb0a77cae89046 not found: ID does not exist"
Jul 28 17:45:23.274961 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.274930    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"]
Jul 28 17:45:23.283144 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.283119    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7dbdbf/test-model-7dbdbf-kserve-58c578948c-7hvn4"]
Jul 28 17:45:23.287639 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.287610    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5714fb99-901d-41ae-a54d-f76d6c0c097a-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:23.287739 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.287647    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5714fb99-901d-41ae-a54d-f76d6c0c097a-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:23.287739 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.287672    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klf59\" (UniqueName: \"kubernetes.io/projected/5714fb99-901d-41ae-a54d-f76d6c0c097a-kube-api-access-klf59\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:23.872441 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:23.872407    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5714fb99-901d-41ae-a54d-f76d6c0c097a" path="/var/lib/kubelet/pods/5714fb99-901d-41ae-a54d-f76d6c0c097a/volumes"
Jul 28 17:45:42.348386 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.348338    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"]
Jul 28 17:45:42.348950 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.348589    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4" podUID="253789e1-f5d7-4881-9cb8-32c352fa18fe" containerName="maas-api" containerID="cri-o://3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0" gracePeriod=30
Jul 28 17:45:42.428825 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.428786    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"]
Jul 28 17:45:42.429039 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.429000    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2" podUID="ecb1ba0b-f6e1-4de8-a482-75cc0220edac" containerName="payload-processing" containerID="cri-o://ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b" gracePeriod=30
Jul 28 17:45:42.436938 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.436911    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"]
Jul 28 17:45:42.437120 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.437099    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf" podUID="2a5bed1d-0ac1-4a0c-8631-f036cac5ea84" containerName="payload-pre-processing" containerID="cri-o://ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a" gracePeriod=30
Jul 28 17:45:42.705507 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.705478    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:45:42.721693 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.721671    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:45:42.727069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.726976    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:45:42.855374 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855278    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sczpf\" (UniqueName: \"kubernetes.io/projected/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-kube-api-access-sczpf\") pod \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\" (UID: \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\") "
Jul 28 17:45:42.855517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855387    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr2fs\" (UniqueName: \"kubernetes.io/projected/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-kube-api-access-sr2fs\") pod \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\" (UID: \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\") "
Jul 28 17:45:42.855517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855412    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253789e1-f5d7-4881-9cb8-32c352fa18fe-openshift-service-ca\") pod \"253789e1-f5d7-4881-9cb8-32c352fa18fe\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") "
Jul 28 17:45:42.855517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855434    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-plugins-config-volume\") pod \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\" (UID: \"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84\") "
Jul 28 17:45:42.855517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855464    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zkjf\" (UniqueName: \"kubernetes.io/projected/253789e1-f5d7-4881-9cb8-32c352fa18fe-kube-api-access-6zkjf\") pod \"253789e1-f5d7-4881-9cb8-32c352fa18fe\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") "
Jul 28 17:45:42.855748 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855535    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls\") pod \"253789e1-f5d7-4881-9cb8-32c352fa18fe\" (UID: \"253789e1-f5d7-4881-9cb8-32c352fa18fe\") "
Jul 28 17:45:42.855748 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855586    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-plugins-config-volume\") pod \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\" (UID: \"ecb1ba0b-f6e1-4de8-a482-75cc0220edac\") "
Jul 28 17:45:42.855872 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855812    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253789e1-f5d7-4881-9cb8-32c352fa18fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "253789e1-f5d7-4881-9cb8-32c352fa18fe" (UID: "253789e1-f5d7-4881-9cb8-32c352fa18fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:42.855872 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.855823    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a5bed1d-0ac1-4a0c-8631-f036cac5ea84" (UID: "2a5bed1d-0ac1-4a0c-8631-f036cac5ea84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:42.856261 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.856238    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecb1ba0b-f6e1-4de8-a482-75cc0220edac" (UID: "ecb1ba0b-f6e1-4de8-a482-75cc0220edac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:42.857645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.857621    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-kube-api-access-sr2fs" (OuterVolumeSpecName: "kube-api-access-sr2fs") pod "ecb1ba0b-f6e1-4de8-a482-75cc0220edac" (UID: "ecb1ba0b-f6e1-4de8-a482-75cc0220edac"). InnerVolumeSpecName "kube-api-access-sr2fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:42.857753 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.857678    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "253789e1-f5d7-4881-9cb8-32c352fa18fe" (UID: "253789e1-f5d7-4881-9cb8-32c352fa18fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:42.857898 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.857879    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253789e1-f5d7-4881-9cb8-32c352fa18fe-kube-api-access-6zkjf" (OuterVolumeSpecName: "kube-api-access-6zkjf") pod "253789e1-f5d7-4881-9cb8-32c352fa18fe" (UID: "253789e1-f5d7-4881-9cb8-32c352fa18fe"). InnerVolumeSpecName "kube-api-access-6zkjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:42.857962 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.857922    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-kube-api-access-sczpf" (OuterVolumeSpecName: "kube-api-access-sczpf") pod "2a5bed1d-0ac1-4a0c-8631-f036cac5ea84" (UID: "2a5bed1d-0ac1-4a0c-8631-f036cac5ea84"). InnerVolumeSpecName "kube-api-access-sczpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:42.957104 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.957075    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr2fs\" (UniqueName: \"kubernetes.io/projected/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-kube-api-access-sr2fs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:42.957104 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.957097    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253789e1-f5d7-4881-9cb8-32c352fa18fe-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:42.957104 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.957107    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:42.957410 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.957115    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zkjf\" (UniqueName: \"kubernetes.io/projected/253789e1-f5d7-4881-9cb8-32c352fa18fe-kube-api-access-6zkjf\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:42.957410 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.957124    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253789e1-f5d7-4881-9cb8-32c352fa18fe-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:42.957410 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.957132    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb1ba0b-f6e1-4de8-a482-75cc0220edac-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:42.957410 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:42.957140    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sczpf\" (UniqueName: \"kubernetes.io/projected/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84-kube-api-access-sczpf\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:45:43.355414 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.355382    2560 generic.go:356] "Generic (PLEG): container finished" podID="ecb1ba0b-f6e1-4de8-a482-75cc0220edac" containerID="ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b" exitCode=0
Jul 28 17:45:43.355878 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.355424    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"
Jul 28 17:45:43.355878 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.355469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2" event={"ID":"ecb1ba0b-f6e1-4de8-a482-75cc0220edac","Type":"ContainerDied","Data":"ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b"}
Jul 28 17:45:43.355878 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.355520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2" event={"ID":"ecb1ba0b-f6e1-4de8-a482-75cc0220edac","Type":"ContainerDied","Data":"fc3cacbf4143614525a5ec8b20e51168f9a8f99eb02706e77ccd7c3881fbbdef"}
Jul 28 17:45:43.355878 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.355544    2560 scope.go:117] "RemoveContainer" containerID="ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b"
Jul 28 17:45:43.356860 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.356840    2560 generic.go:356] "Generic (PLEG): container finished" podID="253789e1-f5d7-4881-9cb8-32c352fa18fe" containerID="3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0" exitCode=0
Jul 28 17:45:43.356954 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.356868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4" event={"ID":"253789e1-f5d7-4881-9cb8-32c352fa18fe","Type":"ContainerDied","Data":"3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0"}
Jul 28 17:45:43.356954 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.356899    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"
Jul 28 17:45:43.356954 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.356912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4" event={"ID":"253789e1-f5d7-4881-9cb8-32c352fa18fe","Type":"ContainerDied","Data":"c51b7f6082ce9fe802adae485b69c6c916e8be75c3dbd7ed837fc22c27f7281d"}
Jul 28 17:45:43.358148 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.358126    2560 generic.go:356] "Generic (PLEG): container finished" podID="2a5bed1d-0ac1-4a0c-8631-f036cac5ea84" containerID="ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a" exitCode=0
Jul 28 17:45:43.358255 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.358199    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"
Jul 28 17:45:43.358255 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.358237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf" event={"ID":"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84","Type":"ContainerDied","Data":"ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a"}
Jul 28 17:45:43.358387 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.358276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf" event={"ID":"2a5bed1d-0ac1-4a0c-8631-f036cac5ea84","Type":"ContainerDied","Data":"a895d0e9cb1f971cb4cb58159a703239ae1eff02450446eed83d11a9ee7d06c4"}
Jul 28 17:45:43.377678 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.377610    2560 scope.go:117] "RemoveContainer" containerID="ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b"
Jul 28 17:45:43.377968 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:45:43.377947    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b\": container with ID starting with ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b not found: ID does not exist" containerID="ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b"
Jul 28 17:45:43.378084 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.377978    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b"} err="failed to get container status \"ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b\": rpc error: code = NotFound desc = could not find container \"ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b\": container with ID starting with ba93efeb2726a025a6da716c04c3323d265659bcd3ba079cbab95c1fb658f51b not found: ID does not exist"
Jul 28 17:45:43.378084 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.378002    2560 scope.go:117] "RemoveContainer" containerID="3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0"
Jul 28 17:45:43.387644 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.387633    2560 scope.go:117] "RemoveContainer" containerID="3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0"
Jul 28 17:45:43.388071 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:45:43.388047    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0\": container with ID starting with 3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0 not found: ID does not exist" containerID="3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0"
Jul 28 17:45:43.388134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.388079    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0"} err="failed to get container status \"3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0\": rpc error: code = NotFound desc = could not find container \"3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0\": container with ID starting with 3ab02f3dad61d0bbd8098602354ab37a4207c418ab6d9561416fa915d29b9cf0 not found: ID does not exist"
Jul 28 17:45:43.388134 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.388096    2560 scope.go:117] "RemoveContainer" containerID="ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a"
Jul 28 17:45:43.392784 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.392763    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"]
Jul 28 17:45:43.395703 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.395682    2560 scope.go:117] "RemoveContainer" containerID="ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a"
Jul 28 17:45:43.395997 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:45:43.395973    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a\": container with ID starting with ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a not found: ID does not exist" containerID="ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a"
Jul 28 17:45:43.396085 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.396004    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a"} err="failed to get container status \"ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a\": rpc error: code = NotFound desc = could not find container \"ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a\": container with ID starting with ca997a1c3eb22960c574a0b0413de39c5e6f96c7097f07fa962eff96cc7e239a not found: ID does not exist"
Jul 28 17:45:43.398262 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.398242    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8796a3-59c67fc444-46ql2"]
Jul 28 17:45:43.409575 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.409554    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"]
Jul 28 17:45:43.414874 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.414856    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8796a3-6979c9c98-rdzk4"]
Jul 28 17:45:43.426629 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.426610    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"]
Jul 28 17:45:43.431082 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.431066    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8796a3-9fcbc86d8-t29zf"]
Jul 28 17:45:43.872300 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.872266    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="253789e1-f5d7-4881-9cb8-32c352fa18fe" path="/var/lib/kubelet/pods/253789e1-f5d7-4881-9cb8-32c352fa18fe/volumes"
Jul 28 17:45:43.872632 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.872618    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5bed1d-0ac1-4a0c-8631-f036cac5ea84" path="/var/lib/kubelet/pods/2a5bed1d-0ac1-4a0c-8631-f036cac5ea84/volumes"
Jul 28 17:45:43.872949 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:43.872937    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb1ba0b-f6e1-4de8-a482-75cc0220edac" path="/var/lib/kubelet/pods/ecb1ba0b-f6e1-4de8-a482-75cc0220edac/volumes"
Jul 28 17:45:57.537322 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537289    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"]
Jul 28 17:45:57.537712 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537692    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431f0b54-569d-49c8-936c-e8c400f47a88" containerName="maas-api"
Jul 28 17:45:57.537712 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537703    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="431f0b54-569d-49c8-936c-e8c400f47a88" containerName="maas-api"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537713    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5bed1d-0ac1-4a0c-8631-f036cac5ea84" containerName="payload-pre-processing"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537720    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5bed1d-0ac1-4a0c-8631-f036cac5ea84" containerName="payload-pre-processing"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537728    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="253789e1-f5d7-4881-9cb8-32c352fa18fe" containerName="maas-api"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537733    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="253789e1-f5d7-4881-9cb8-32c352fa18fe" containerName="maas-api"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537738    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5714fb99-901d-41ae-a54d-f76d6c0c097a" containerName="main"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537743    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5714fb99-901d-41ae-a54d-f76d6c0c097a" containerName="main"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537756    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34d8187-0213-4671-b7da-9e04bc92f00a" containerName="payload-processing"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537761    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34d8187-0213-4671-b7da-9e04bc92f00a" containerName="payload-processing"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537771    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b9d714-962e-4fc4-b251-1d9151aa3488" containerName="payload-pre-processing"
Jul 28 17:45:57.537783 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537777    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b9d714-962e-4fc4-b251-1d9151aa3488" containerName="payload-pre-processing"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537798    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb1ba0b-f6e1-4de8-a482-75cc0220edac" containerName="payload-processing"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537804    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb1ba0b-f6e1-4de8-a482-75cc0220edac" containerName="payload-processing"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537857    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb1ba0b-f6e1-4de8-a482-75cc0220edac" containerName="payload-processing"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537864    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b9d714-962e-4fc4-b251-1d9151aa3488" containerName="payload-pre-processing"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537871    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5bed1d-0ac1-4a0c-8631-f036cac5ea84" containerName="payload-pre-processing"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537876    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34d8187-0213-4671-b7da-9e04bc92f00a" containerName="payload-processing"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537881    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="431f0b54-569d-49c8-936c-e8c400f47a88" containerName="maas-api"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537887    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="253789e1-f5d7-4881-9cb8-32c352fa18fe" containerName="maas-api"
Jul 28 17:45:57.538069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.537892    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5714fb99-901d-41ae-a54d-f76d6c0c097a" containerName="main"
Jul 28 17:45:57.540789 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.540772    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.544245 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.544217    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:45:57.544400 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.544383    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:45:57.544678 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.544664    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:45:57.550001 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.549982    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7869f9-serving-cert\""
Jul 28 17:45:57.558418 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.558395    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"]
Jul 28 17:45:57.679408 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.679376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqwn\" (UniqueName: \"kubernetes.io/projected/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-kube-api-access-srqwn\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.679557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.679465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-maas-api-tls\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.679557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.679494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-openshift-service-ca\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.780414 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.780388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqwn\" (UniqueName: \"kubernetes.io/projected/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-kube-api-access-srqwn\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.780536 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.780455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-maas-api-tls\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.780536 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.780495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-openshift-service-ca\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.781175 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.781154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-openshift-service-ca\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.782785 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.782762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-maas-api-tls\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.800617 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.800597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqwn\" (UniqueName: \"kubernetes.io/projected/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-kube-api-access-srqwn\") pod \"maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:57.850197 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:57.850174    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:58.004160 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.003951    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"]
Jul 28 17:45:58.006418 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:45:58.006392    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ca48fc_12c3_4a6f_848d_a08e8d02ed50.slice/crio-cd87f9bb0c8d1ddd8e92450bc807d5d008846e6d03cfe3f2a13ebdbd3068fe33 WatchSource:0}: Error finding container cd87f9bb0c8d1ddd8e92450bc807d5d008846e6d03cfe3f2a13ebdbd3068fe33: Status 404 returned error can't find the container with id cd87f9bb0c8d1ddd8e92450bc807d5d008846e6d03cfe3f2a13ebdbd3068fe33
Jul 28 17:45:58.007847 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.007824    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:45:58.431374 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.431319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588" event={"ID":"27ca48fc-12c3-4a6f-848d-a08e8d02ed50","Type":"ContainerStarted","Data":"cd87f9bb0c8d1ddd8e92450bc807d5d008846e6d03cfe3f2a13ebdbd3068fe33"}
Jul 28 17:45:58.482294 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.482264    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"]
Jul 28 17:45:58.486692 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.486676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:58.490537 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.490521    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7869f9\""
Jul 28 17:45:58.491144 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.491126    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7869f9-dockercfg-6mblh\""
Jul 28 17:45:58.500897 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.500877    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"]
Jul 28 17:45:58.544627 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.544608    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"]
Jul 28 17:45:58.548981 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.548967    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:58.559277 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.559254    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"]
Jul 28 17:45:58.587208 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.587180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fecb1970-4f1e-437b-9598-dd0c49d6a027-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws\" (UID: \"fecb1970-4f1e-437b-9598-dd0c49d6a027\") " pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:58.587305 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.587287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n479d\" (UniqueName: \"kubernetes.io/projected/fecb1970-4f1e-437b-9598-dd0c49d6a027-kube-api-access-n479d\") pod \"payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws\" (UID: \"fecb1970-4f1e-437b-9598-dd0c49d6a027\") " pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:58.688476 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.688407    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6kxb\" (UniqueName: \"kubernetes.io/projected/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-kube-api-access-l6kxb\") pod \"payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578\" (UID: \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:58.688476 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.688471    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fecb1970-4f1e-437b-9598-dd0c49d6a027-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws\" (UID: \"fecb1970-4f1e-437b-9598-dd0c49d6a027\") " pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:58.688643 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.688521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578\" (UID: \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:58.688643 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.688589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n479d\" (UniqueName: \"kubernetes.io/projected/fecb1970-4f1e-437b-9598-dd0c49d6a027-kube-api-access-n479d\") pod \"payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws\" (UID: \"fecb1970-4f1e-437b-9598-dd0c49d6a027\") " pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:58.689110 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.689093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fecb1970-4f1e-437b-9598-dd0c49d6a027-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws\" (UID: \"fecb1970-4f1e-437b-9598-dd0c49d6a027\") " pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:58.698673 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.698655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n479d\" (UniqueName: \"kubernetes.io/projected/fecb1970-4f1e-437b-9598-dd0c49d6a027-kube-api-access-n479d\") pod \"payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws\" (UID: \"fecb1970-4f1e-437b-9598-dd0c49d6a027\") " pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:58.789445 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.789419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578\" (UID: \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:58.789557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.789492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6kxb\" (UniqueName: \"kubernetes.io/projected/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-kube-api-access-l6kxb\") pod \"payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578\" (UID: \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:58.790248 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.790225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578\" (UID: \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:58.797933 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.797912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:58.798686 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.798669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6kxb\" (UniqueName: \"kubernetes.io/projected/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-kube-api-access-l6kxb\") pod \"payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578\" (UID: \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:58.873216 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.873190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:58.931017 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:58.930992    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"]
Jul 28 17:45:58.932949 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:45:58.932905    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecb1970_4f1e_437b_9598_dd0c49d6a027.slice/crio-0b1482197ae02bf325a3900d5fe32e82c2c16c082a901ee90f6c5ae862f7776b WatchSource:0}: Error finding container 0b1482197ae02bf325a3900d5fe32e82c2c16c082a901ee90f6c5ae862f7776b: Status 404 returned error can't find the container with id 0b1482197ae02bf325a3900d5fe32e82c2c16c082a901ee90f6c5ae862f7776b
Jul 28 17:45:59.019495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.019474    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"]
Jul 28 17:45:59.022079 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:45:59.022056    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe54260_bcc7_4d12_9cb6_c491894a7d2c.slice/crio-a59bf170ca1b1be2ca4b5b01d3d85b3f7b35544d89b8ae6e3ee942129167e0bd WatchSource:0}: Error finding container a59bf170ca1b1be2ca4b5b01d3d85b3f7b35544d89b8ae6e3ee942129167e0bd: Status 404 returned error can't find the container with id a59bf170ca1b1be2ca4b5b01d3d85b3f7b35544d89b8ae6e3ee942129167e0bd
Jul 28 17:45:59.438603 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.438559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578" event={"ID":"4fe54260-bcc7-4d12-9cb6-c491894a7d2c","Type":"ContainerStarted","Data":"7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60"}
Jul 28 17:45:59.438762 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.438631    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:45:59.438762 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.438658    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578" event={"ID":"4fe54260-bcc7-4d12-9cb6-c491894a7d2c","Type":"ContainerStarted","Data":"a59bf170ca1b1be2ca4b5b01d3d85b3f7b35544d89b8ae6e3ee942129167e0bd"}
Jul 28 17:45:59.440163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.440138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588" event={"ID":"27ca48fc-12c3-4a6f-848d-a08e8d02ed50","Type":"ContainerStarted","Data":"8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497"}
Jul 28 17:45:59.440293 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.440277    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:45:59.441556 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.441532    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws" event={"ID":"fecb1970-4f1e-437b-9598-dd0c49d6a027","Type":"ContainerStarted","Data":"02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4"}
Jul 28 17:45:59.441635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.441570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:45:59.441635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.441588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws" event={"ID":"fecb1970-4f1e-437b-9598-dd0c49d6a027","Type":"ContainerStarted","Data":"0b1482197ae02bf325a3900d5fe32e82c2c16c082a901ee90f6c5ae862f7776b"}
Jul 28 17:45:59.459654 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.459614    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578" podStartSLOduration=1.459601814 podStartE2EDuration="1.459601814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:59.458085573 +0000 UTC m=+2230.130413493" watchObservedRunningTime="2026-07-28 17:45:59.459601814 +0000 UTC m=+2230.131929727"
Jul 28 17:45:59.481124 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.481087    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws" podStartSLOduration=1.481077466 podStartE2EDuration="1.481077466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:59.479814685 +0000 UTC m=+2230.152142604" watchObservedRunningTime="2026-07-28 17:45:59.481077466 +0000 UTC m=+2230.153405385"
Jul 28 17:45:59.499064 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:45:59.499012    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588" podStartSLOduration=1.988675615 podStartE2EDuration="2.498996763s" totalImagesPullingTime="510.321148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:59.498504309 +0000 UTC m=+2230.170832229" watchObservedRunningTime="2026-07-28 17:45:59.498996763 +0000 UTC m=+2230.171324685"
Jul 28 17:46:05.451149 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.451116    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:46:05.601635 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.601596    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"]
Jul 28 17:46:05.605445 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.605425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:05.608586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.608567    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-29d9e9-serving-cert\""
Jul 28 17:46:05.615228 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.615206    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"]
Jul 28 17:46:05.750018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.749953    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:05.750147 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.750018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsb8r\" (UniqueName: \"kubernetes.io/projected/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-kube-api-access-lsb8r\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:05.750147 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.750101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-openshift-service-ca\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:05.850992 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.850967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsb8r\" (UniqueName: \"kubernetes.io/projected/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-kube-api-access-lsb8r\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:05.851123 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.851005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-openshift-service-ca\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:05.851123 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.851040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:05.851242 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:46:05.851133    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-serving-cert: secret "maas-api-e2e-api-b-29d9e9-serving-cert" not found
Jul 28 17:46:05.851242 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:46:05.851182    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls podName:7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8 nodeName:}" failed. No retries permitted until 2026-07-28 17:46:06.351166683 +0000 UTC m=+2237.023494585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls") pod "maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx" (UID: "7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8") : secret "maas-api-e2e-api-b-29d9e9-serving-cert" not found
Jul 28 17:46:05.851618 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.851600    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-openshift-service-ca\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:05.860398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:05.860380    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsb8r\" (UniqueName: \"kubernetes.io/projected/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-kube-api-access-lsb8r\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:06.355597 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.355561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:06.358009 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.357986    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls\") pod \"maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:06.488787 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.488747    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"]
Jul 28 17:46:06.492588 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.492572    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:06.495300 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.495281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-29d9e9-dockercfg-xljng\""
Jul 28 17:46:06.495404 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.495281    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-29d9e9\""
Jul 28 17:46:06.502237 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.502215    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"]
Jul 28 17:46:06.517176 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.517160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:06.555164 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.555132    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"]
Jul 28 17:46:06.559908 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.559886    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:06.571648 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.571626    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"]
Jul 28 17:46:06.657118 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.657092    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"]
Jul 28 17:46:06.659100 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.659073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8vv\" (UniqueName: \"kubernetes.io/projected/c98aa562-ff57-42fc-a947-f58e906a6f25-kube-api-access-7v8vv\") pod \"payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl\" (UID: \"c98aa562-ff57-42fc-a947-f58e906a6f25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:06.659169 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.659153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22qpd\" (UniqueName: \"kubernetes.io/projected/bdbfeab3-497b-4d9a-8729-06746b855fef-kube-api-access-22qpd\") pod \"payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2\" (UID: \"bdbfeab3-497b-4d9a-8729-06746b855fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:06.659220 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.659188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98aa562-ff57-42fc-a947-f58e906a6f25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl\" (UID: \"c98aa562-ff57-42fc-a947-f58e906a6f25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:06.659260 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.659227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbfeab3-497b-4d9a-8729-06746b855fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2\" (UID: \"bdbfeab3-497b-4d9a-8729-06746b855fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:06.659788 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:46:06.659769    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9bedc6_adc0_4e2c_9f3a_920d68bd65c8.slice/crio-1cf8b00a76a577dcbb093980b83d0fe30ace0e3dec00283512250048dd00b126 WatchSource:0}: Error finding container 1cf8b00a76a577dcbb093980b83d0fe30ace0e3dec00283512250048dd00b126: Status 404 returned error can't find the container with id 1cf8b00a76a577dcbb093980b83d0fe30ace0e3dec00283512250048dd00b126
Jul 28 17:46:06.759985 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.759951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8vv\" (UniqueName: \"kubernetes.io/projected/c98aa562-ff57-42fc-a947-f58e906a6f25-kube-api-access-7v8vv\") pod \"payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl\" (UID: \"c98aa562-ff57-42fc-a947-f58e906a6f25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:06.760144 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.760002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22qpd\" (UniqueName: \"kubernetes.io/projected/bdbfeab3-497b-4d9a-8729-06746b855fef-kube-api-access-22qpd\") pod \"payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2\" (UID: \"bdbfeab3-497b-4d9a-8729-06746b855fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:06.760255 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.760220    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98aa562-ff57-42fc-a947-f58e906a6f25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl\" (UID: \"c98aa562-ff57-42fc-a947-f58e906a6f25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:06.760300 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.760285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbfeab3-497b-4d9a-8729-06746b855fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2\" (UID: \"bdbfeab3-497b-4d9a-8729-06746b855fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:06.760897 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.760878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98aa562-ff57-42fc-a947-f58e906a6f25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl\" (UID: \"c98aa562-ff57-42fc-a947-f58e906a6f25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:06.760969 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.760951    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbfeab3-497b-4d9a-8729-06746b855fef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2\" (UID: \"bdbfeab3-497b-4d9a-8729-06746b855fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:06.776600 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.776574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22qpd\" (UniqueName: \"kubernetes.io/projected/bdbfeab3-497b-4d9a-8729-06746b855fef-kube-api-access-22qpd\") pod \"payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2\" (UID: \"bdbfeab3-497b-4d9a-8729-06746b855fef\") " pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:06.776724 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.776668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8vv\" (UniqueName: \"kubernetes.io/projected/c98aa562-ff57-42fc-a947-f58e906a6f25-kube-api-access-7v8vv\") pod \"payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl\" (UID: \"c98aa562-ff57-42fc-a947-f58e906a6f25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:06.803470 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.803446    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:06.876199 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.876170    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:06.960703 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:06.960676    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"]
Jul 28 17:46:06.964032 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:46:06.964006    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbfeab3_497b_4d9a_8729_06746b855fef.slice/crio-7f04d18c4da781738c0d6a71acb426e19eb4bfb42cb6b809aa303b2402261378 WatchSource:0}: Error finding container 7f04d18c4da781738c0d6a71acb426e19eb4bfb42cb6b809aa303b2402261378: Status 404 returned error can't find the container with id 7f04d18c4da781738c0d6a71acb426e19eb4bfb42cb6b809aa303b2402261378
Jul 28 17:46:07.020122 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.020100    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"]
Jul 28 17:46:07.022060 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:46:07.022037    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98aa562_ff57_42fc_a947_f58e906a6f25.slice/crio-638eea29a3c5793dfd462b0b5aa04abe1b8b77a557dadaf88454cd056696310c WatchSource:0}: Error finding container 638eea29a3c5793dfd462b0b5aa04abe1b8b77a557dadaf88454cd056696310c: Status 404 returned error can't find the container with id 638eea29a3c5793dfd462b0b5aa04abe1b8b77a557dadaf88454cd056696310c
Jul 28 17:46:07.478950 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.478915    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx" event={"ID":"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8","Type":"ContainerStarted","Data":"1cf8b00a76a577dcbb093980b83d0fe30ace0e3dec00283512250048dd00b126"}
Jul 28 17:46:07.480396 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.480362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl" event={"ID":"c98aa562-ff57-42fc-a947-f58e906a6f25","Type":"ContainerStarted","Data":"871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f"}
Jul 28 17:46:07.480545 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.480422    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl" event={"ID":"c98aa562-ff57-42fc-a947-f58e906a6f25","Type":"ContainerStarted","Data":"638eea29a3c5793dfd462b0b5aa04abe1b8b77a557dadaf88454cd056696310c"}
Jul 28 17:46:07.480545 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.480454    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:07.481847 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.481829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2" event={"ID":"bdbfeab3-497b-4d9a-8729-06746b855fef","Type":"ContainerStarted","Data":"ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025"}
Jul 28 17:46:07.481913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.481864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2" event={"ID":"bdbfeab3-497b-4d9a-8729-06746b855fef","Type":"ContainerStarted","Data":"7f04d18c4da781738c0d6a71acb426e19eb4bfb42cb6b809aa303b2402261378"}
Jul 28 17:46:07.481913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.481883    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:07.500082 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.500039    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl" podStartSLOduration=1.500024595 podStartE2EDuration="1.500024595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:07.498120761 +0000 UTC m=+2238.170448693" watchObservedRunningTime="2026-07-28 17:46:07.500024595 +0000 UTC m=+2238.172352511"
Jul 28 17:46:07.516665 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:07.516624    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2" podStartSLOduration=1.5166107009999998 podStartE2EDuration="1.516610701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:07.514770365 +0000 UTC m=+2238.187098299" watchObservedRunningTime="2026-07-28 17:46:07.516610701 +0000 UTC m=+2238.188938620"
Jul 28 17:46:08.488107 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:08.488069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx" event={"ID":"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8","Type":"ContainerStarted","Data":"9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5"}
Jul 28 17:46:08.488355 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:08.488322    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:08.510464 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:08.510415    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx" podStartSLOduration=2.661077445 podStartE2EDuration="3.510401544s" totalImagesPullingTime="849.324099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:08.508771817 +0000 UTC m=+2239.181099738" watchObservedRunningTime="2026-07-28 17:46:08.510401544 +0000 UTC m=+2239.182729463"
Jul 28 17:46:10.447203 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:10.447176    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:46:10.447579 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:10.447356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:46:14.498360 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:14.498268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:46:18.490445 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:18.490415    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:46:18.490846 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:18.490480    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:46:33.812269 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:33.812234    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"]
Jul 28 17:46:33.812812 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:33.812517    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588" podUID="27ca48fc-12c3-4a6f-848d-a08e8d02ed50" containerName="maas-api" containerID="cri-o://8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497" gracePeriod=30
Jul 28 17:46:33.875653 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:33.875620    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"]
Jul 28 17:46:33.875841 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:33.875696    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"]
Jul 28 17:46:33.876714 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:33.875894    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws" podUID="fecb1970-4f1e-437b-9598-dd0c49d6a027" containerName="payload-processing" containerID="cri-o://02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4" gracePeriod=30
Jul 28 17:46:33.876934 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:33.876807    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578" podUID="4fe54260-bcc7-4d12-9cb6-c491894a7d2c" containerName="payload-pre-processing" containerID="cri-o://7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60" gracePeriod=30
Jul 28 17:46:34.129365 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.129329    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:46:34.164183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.164156    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:46:34.173878 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.173857    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:46:34.179191 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.179173    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-openshift-service-ca\") pod \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") "
Jul 28 17:46:34.179273 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.179240    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srqwn\" (UniqueName: \"kubernetes.io/projected/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-kube-api-access-srqwn\") pod \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") "
Jul 28 17:46:34.179316 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.179275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-maas-api-tls\") pod \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\" (UID: \"27ca48fc-12c3-4a6f-848d-a08e8d02ed50\") "
Jul 28 17:46:34.179572 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.179551    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27ca48fc-12c3-4a6f-848d-a08e8d02ed50" (UID: "27ca48fc-12c3-4a6f-848d-a08e8d02ed50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:34.181177 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.181159    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27ca48fc-12c3-4a6f-848d-a08e8d02ed50" (UID: "27ca48fc-12c3-4a6f-848d-a08e8d02ed50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:34.181236 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.181217    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-kube-api-access-srqwn" (OuterVolumeSpecName: "kube-api-access-srqwn") pod "27ca48fc-12c3-4a6f-848d-a08e8d02ed50" (UID: "27ca48fc-12c3-4a6f-848d-a08e8d02ed50"). InnerVolumeSpecName "kube-api-access-srqwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:34.280567 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.280534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6kxb\" (UniqueName: \"kubernetes.io/projected/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-kube-api-access-l6kxb\") pod \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\" (UID: \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\") "
Jul 28 17:46:34.280567 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.280574    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fecb1970-4f1e-437b-9598-dd0c49d6a027-plugins-config-volume\") pod \"fecb1970-4f1e-437b-9598-dd0c49d6a027\" (UID: \"fecb1970-4f1e-437b-9598-dd0c49d6a027\") "
Jul 28 17:46:34.280792 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.280641    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-plugins-config-volume\") pod \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\" (UID: \"4fe54260-bcc7-4d12-9cb6-c491894a7d2c\") "
Jul 28 17:46:34.280792 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.280712    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n479d\" (UniqueName: \"kubernetes.io/projected/fecb1970-4f1e-437b-9598-dd0c49d6a027-kube-api-access-n479d\") pod \"fecb1970-4f1e-437b-9598-dd0c49d6a027\" (UID: \"fecb1970-4f1e-437b-9598-dd0c49d6a027\") "
Jul 28 17:46:34.281053 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.281026    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecb1970-4f1e-437b-9598-dd0c49d6a027-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fecb1970-4f1e-437b-9598-dd0c49d6a027" (UID: "fecb1970-4f1e-437b-9598-dd0c49d6a027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:34.281148 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.281039    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:46:34.281148 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.281074    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srqwn\" (UniqueName: \"kubernetes.io/projected/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-kube-api-access-srqwn\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:46:34.281148 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.281084    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ca48fc-12c3-4a6f-848d-a08e8d02ed50-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:46:34.281148 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.281049    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fe54260-bcc7-4d12-9cb6-c491894a7d2c" (UID: "4fe54260-bcc7-4d12-9cb6-c491894a7d2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:34.282717 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.282698    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-kube-api-access-l6kxb" (OuterVolumeSpecName: "kube-api-access-l6kxb") pod "4fe54260-bcc7-4d12-9cb6-c491894a7d2c" (UID: "4fe54260-bcc7-4d12-9cb6-c491894a7d2c"). InnerVolumeSpecName "kube-api-access-l6kxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:34.282777 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.282746    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecb1970-4f1e-437b-9598-dd0c49d6a027-kube-api-access-n479d" (OuterVolumeSpecName: "kube-api-access-n479d") pod "fecb1970-4f1e-437b-9598-dd0c49d6a027" (UID: "fecb1970-4f1e-437b-9598-dd0c49d6a027"). InnerVolumeSpecName "kube-api-access-n479d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:34.381774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.381698    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n479d\" (UniqueName: \"kubernetes.io/projected/fecb1970-4f1e-437b-9598-dd0c49d6a027-kube-api-access-n479d\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:46:34.381774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.381724    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6kxb\" (UniqueName: \"kubernetes.io/projected/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-kube-api-access-l6kxb\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:46:34.381774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.381733    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fecb1970-4f1e-437b-9598-dd0c49d6a027-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:46:34.381774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.381743    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe54260-bcc7-4d12-9cb6-c491894a7d2c-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:46:34.617841 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.617799    2560 generic.go:356] "Generic (PLEG): container finished" podID="4fe54260-bcc7-4d12-9cb6-c491894a7d2c" containerID="7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60" exitCode=0
Jul 28 17:46:34.618035 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.617848    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"
Jul 28 17:46:34.618035 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.617877    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578" event={"ID":"4fe54260-bcc7-4d12-9cb6-c491894a7d2c","Type":"ContainerDied","Data":"7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60"}
Jul 28 17:46:34.618035 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.617931    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578" event={"ID":"4fe54260-bcc7-4d12-9cb6-c491894a7d2c","Type":"ContainerDied","Data":"a59bf170ca1b1be2ca4b5b01d3d85b3f7b35544d89b8ae6e3ee942129167e0bd"}
Jul 28 17:46:34.618035 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.617960    2560 scope.go:117] "RemoveContainer" containerID="7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60"
Jul 28 17:46:34.619328 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.619307    2560 generic.go:356] "Generic (PLEG): container finished" podID="27ca48fc-12c3-4a6f-848d-a08e8d02ed50" containerID="8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497" exitCode=0
Jul 28 17:46:34.619442 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.619376    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"
Jul 28 17:46:34.619442 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.619389    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588" event={"ID":"27ca48fc-12c3-4a6f-848d-a08e8d02ed50","Type":"ContainerDied","Data":"8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497"}
Jul 28 17:46:34.619442 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.619428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588" event={"ID":"27ca48fc-12c3-4a6f-848d-a08e8d02ed50","Type":"ContainerDied","Data":"cd87f9bb0c8d1ddd8e92450bc807d5d008846e6d03cfe3f2a13ebdbd3068fe33"}
Jul 28 17:46:34.620884 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.620864    2560 generic.go:356] "Generic (PLEG): container finished" podID="fecb1970-4f1e-437b-9598-dd0c49d6a027" containerID="02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4" exitCode=0
Jul 28 17:46:34.621003 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.620907    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"
Jul 28 17:46:34.621003 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.620928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws" event={"ID":"fecb1970-4f1e-437b-9598-dd0c49d6a027","Type":"ContainerDied","Data":"02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4"}
Jul 28 17:46:34.621003 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.620959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws" event={"ID":"fecb1970-4f1e-437b-9598-dd0c49d6a027","Type":"ContainerDied","Data":"0b1482197ae02bf325a3900d5fe32e82c2c16c082a901ee90f6c5ae862f7776b"}
Jul 28 17:46:34.627693 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.627667    2560 scope.go:117] "RemoveContainer" containerID="7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60"
Jul 28 17:46:34.627965 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:46:34.627949    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60\": container with ID starting with 7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60 not found: ID does not exist" containerID="7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60"
Jul 28 17:46:34.628030 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.627971    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60"} err="failed to get container status \"7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60\": rpc error: code = NotFound desc = could not find container \"7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60\": container with ID starting with 7813264f1fcd633ac989cb74d6e214c3da52f08258afc8d74c2a9eff8b9c8d60 not found: ID does not exist"
Jul 28 17:46:34.628030 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.627986    2560 scope.go:117] "RemoveContainer" containerID="8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497"
Jul 28 17:46:34.637053 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.637037    2560 scope.go:117] "RemoveContainer" containerID="8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497"
Jul 28 17:46:34.637304 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:46:34.637289    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497\": container with ID starting with 8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497 not found: ID does not exist" containerID="8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497"
Jul 28 17:46:34.637370 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.637311    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497"} err="failed to get container status \"8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497\": rpc error: code = NotFound desc = could not find container \"8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497\": container with ID starting with 8592086a5221b4ea043f2e599ce4600160978bbcbc5fa348616ef8b4f1ae6497 not found: ID does not exist"
Jul 28 17:46:34.637370 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.637325    2560 scope.go:117] "RemoveContainer" containerID="02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4"
Jul 28 17:46:34.644883 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.644864    2560 scope.go:117] "RemoveContainer" containerID="02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4"
Jul 28 17:46:34.645252 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:46:34.645227    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4\": container with ID starting with 02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4 not found: ID does not exist" containerID="02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4"
Jul 28 17:46:34.645366 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.645262    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4"} err="failed to get container status \"02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4\": rpc error: code = NotFound desc = could not find container \"02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4\": container with ID starting with 02a6f8d0ab3525deef51fde9cd3558bbc1266195e752fdf6bb3cc3e704869ef4 not found: ID does not exist"
Jul 28 17:46:34.646940 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.646921    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"]
Jul 28 17:46:34.650017 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.649996    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7869f9-5fcb8c757b-mm578"]
Jul 28 17:46:34.663533 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.663507    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"]
Jul 28 17:46:34.666686 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.666668    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7869f9-5b45d9bfcd-mnqws"]
Jul 28 17:46:34.678745 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.678722    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"]
Jul 28 17:46:34.681562 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:34.681546    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7869f9-cf95ccc5c-wl588"]
Jul 28 17:46:35.872640 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:35.872609    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27ca48fc-12c3-4a6f-848d-a08e8d02ed50" path="/var/lib/kubelet/pods/27ca48fc-12c3-4a6f-848d-a08e8d02ed50/volumes"
Jul 28 17:46:35.873018 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:35.872990    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe54260-bcc7-4d12-9cb6-c491894a7d2c" path="/var/lib/kubelet/pods/4fe54260-bcc7-4d12-9cb6-c491894a7d2c/volumes"
Jul 28 17:46:35.873314 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:46:35.873302    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fecb1970-4f1e-437b-9598-dd0c49d6a027" path="/var/lib/kubelet/pods/fecb1970-4f1e-437b-9598-dd0c49d6a027/volumes"
Jul 28 17:47:01.819683 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:01.819647    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"]
Jul 28 17:47:01.820066 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:01.819851    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx" podUID="7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" containerName="maas-api" containerID="cri-o://9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5" gracePeriod=30
Jul 28 17:47:01.878228 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:01.878199    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"]
Jul 28 17:47:01.878461 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:01.878417    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2" podUID="bdbfeab3-497b-4d9a-8729-06746b855fef" containerName="payload-processing" containerID="cri-o://ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025" gracePeriod=30
Jul 28 17:47:01.895234 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:01.895209    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"]
Jul 28 17:47:01.895445 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:01.895424    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl" podUID="c98aa562-ff57-42fc-a947-f58e906a6f25" containerName="payload-pre-processing" containerID="cri-o://871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f" gracePeriod=30
Jul 28 17:47:02.173069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.173049    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:47:02.176226 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.176205    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:47:02.179733 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.179711    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:47:02.241520 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.241494    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98aa562-ff57-42fc-a947-f58e906a6f25-plugins-config-volume\") pod \"c98aa562-ff57-42fc-a947-f58e906a6f25\" (UID: \"c98aa562-ff57-42fc-a947-f58e906a6f25\") "
Jul 28 17:47:02.241653 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.241571    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbfeab3-497b-4d9a-8729-06746b855fef-plugins-config-volume\") pod \"bdbfeab3-497b-4d9a-8729-06746b855fef\" (UID: \"bdbfeab3-497b-4d9a-8729-06746b855fef\") "
Jul 28 17:47:02.241653 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.241618    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsb8r\" (UniqueName: \"kubernetes.io/projected/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-kube-api-access-lsb8r\") pod \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") "
Jul 28 17:47:02.241807 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.241677    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22qpd\" (UniqueName: \"kubernetes.io/projected/bdbfeab3-497b-4d9a-8729-06746b855fef-kube-api-access-22qpd\") pod \"bdbfeab3-497b-4d9a-8729-06746b855fef\" (UID: \"bdbfeab3-497b-4d9a-8729-06746b855fef\") "
Jul 28 17:47:02.241807 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.241706    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v8vv\" (UniqueName: \"kubernetes.io/projected/c98aa562-ff57-42fc-a947-f58e906a6f25-kube-api-access-7v8vv\") pod \"c98aa562-ff57-42fc-a947-f58e906a6f25\" (UID: \"c98aa562-ff57-42fc-a947-f58e906a6f25\") "
Jul 28 17:47:02.241807 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.241787    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls\") pod \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") "
Jul 28 17:47:02.241972 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.241803    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98aa562-ff57-42fc-a947-f58e906a6f25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c98aa562-ff57-42fc-a947-f58e906a6f25" (UID: "c98aa562-ff57-42fc-a947-f58e906a6f25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:02.241972 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.241896    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbfeab3-497b-4d9a-8729-06746b855fef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bdbfeab3-497b-4d9a-8729-06746b855fef" (UID: "bdbfeab3-497b-4d9a-8729-06746b855fef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:02.242136 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.242117    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98aa562-ff57-42fc-a947-f58e906a6f25-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:47:02.242203 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.242140    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbfeab3-497b-4d9a-8729-06746b855fef-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:47:02.243816 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.243780    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-kube-api-access-lsb8r" (OuterVolumeSpecName: "kube-api-access-lsb8r") pod "7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" (UID: "7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8"). InnerVolumeSpecName "kube-api-access-lsb8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:02.243912 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.243884    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" (UID: "7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:02.244173 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.244154    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbfeab3-497b-4d9a-8729-06746b855fef-kube-api-access-22qpd" (OuterVolumeSpecName: "kube-api-access-22qpd") pod "bdbfeab3-497b-4d9a-8729-06746b855fef" (UID: "bdbfeab3-497b-4d9a-8729-06746b855fef"). InnerVolumeSpecName "kube-api-access-22qpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:02.244235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.244186    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98aa562-ff57-42fc-a947-f58e906a6f25-kube-api-access-7v8vv" (OuterVolumeSpecName: "kube-api-access-7v8vv") pod "c98aa562-ff57-42fc-a947-f58e906a6f25" (UID: "c98aa562-ff57-42fc-a947-f58e906a6f25"). InnerVolumeSpecName "kube-api-access-7v8vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:02.342462 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.342437    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-openshift-service-ca\") pod \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\" (UID: \"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8\") "
Jul 28 17:47:02.342646 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.342633    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsb8r\" (UniqueName: \"kubernetes.io/projected/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-kube-api-access-lsb8r\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:47:02.342693 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.342648    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22qpd\" (UniqueName: \"kubernetes.io/projected/bdbfeab3-497b-4d9a-8729-06746b855fef-kube-api-access-22qpd\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:47:02.342693 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.342659    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8vv\" (UniqueName: \"kubernetes.io/projected/c98aa562-ff57-42fc-a947-f58e906a6f25-kube-api-access-7v8vv\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:47:02.342693 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.342667    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:47:02.342793 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.342729    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" (UID: "7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:02.443433 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.443379    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:47:02.758690 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.758572    2560 generic.go:356] "Generic (PLEG): container finished" podID="bdbfeab3-497b-4d9a-8729-06746b855fef" containerID="ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025" exitCode=0
Jul 28 17:47:02.758690 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.758656    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"
Jul 28 17:47:02.758690 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.758662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2" event={"ID":"bdbfeab3-497b-4d9a-8729-06746b855fef","Type":"ContainerDied","Data":"ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025"}
Jul 28 17:47:02.758955 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.758750    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2" event={"ID":"bdbfeab3-497b-4d9a-8729-06746b855fef","Type":"ContainerDied","Data":"7f04d18c4da781738c0d6a71acb426e19eb4bfb42cb6b809aa303b2402261378"}
Jul 28 17:47:02.758955 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.758780    2560 scope.go:117] "RemoveContainer" containerID="ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025"
Jul 28 17:47:02.760027 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.760005    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" containerID="9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5" exitCode=0
Jul 28 17:47:02.760132 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.760033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx" event={"ID":"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8","Type":"ContainerDied","Data":"9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5"}
Jul 28 17:47:02.760132 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.760060    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"
Jul 28 17:47:02.760132 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.760094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx" event={"ID":"7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8","Type":"ContainerDied","Data":"1cf8b00a76a577dcbb093980b83d0fe30ace0e3dec00283512250048dd00b126"}
Jul 28 17:47:02.761520 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.761493    2560 generic.go:356] "Generic (PLEG): container finished" podID="c98aa562-ff57-42fc-a947-f58e906a6f25" containerID="871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f" exitCode=0
Jul 28 17:47:02.761631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.761536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl" event={"ID":"c98aa562-ff57-42fc-a947-f58e906a6f25","Type":"ContainerDied","Data":"871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f"}
Jul 28 17:47:02.761631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.761547    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"
Jul 28 17:47:02.761631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.761575    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl" event={"ID":"c98aa562-ff57-42fc-a947-f58e906a6f25","Type":"ContainerDied","Data":"638eea29a3c5793dfd462b0b5aa04abe1b8b77a557dadaf88454cd056696310c"}
Jul 28 17:47:02.770714 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.770491    2560 scope.go:117] "RemoveContainer" containerID="ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025"
Jul 28 17:47:02.771281 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:47:02.771255    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025\": container with ID starting with ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025 not found: ID does not exist" containerID="ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025"
Jul 28 17:47:02.771411 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.771289    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025"} err="failed to get container status \"ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025\": rpc error: code = NotFound desc = could not find container \"ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025\": container with ID starting with ae0f06cfe5e6e0ef32cbaf677dab3961ac24871e74b390bd2549608c029f6025 not found: ID does not exist"
Jul 28 17:47:02.771411 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.771314    2560 scope.go:117] "RemoveContainer" containerID="9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5"
Jul 28 17:47:02.780071 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.780056    2560 scope.go:117] "RemoveContainer" containerID="9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5"
Jul 28 17:47:02.780295 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:47:02.780275    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5\": container with ID starting with 9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5 not found: ID does not exist" containerID="9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5"
Jul 28 17:47:02.780386 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.780300    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5"} err="failed to get container status \"9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5\": rpc error: code = NotFound desc = could not find container \"9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5\": container with ID starting with 9af652cec2e59d8227e1cb30f2b109e56cd06d13c79506d6223564163ad221b5 not found: ID does not exist"
Jul 28 17:47:02.780386 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.780316    2560 scope.go:117] "RemoveContainer" containerID="871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f"
Jul 28 17:47:02.787409 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.787393    2560 scope.go:117] "RemoveContainer" containerID="871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f"
Jul 28 17:47:02.787669 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:47:02.787648    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f\": container with ID starting with 871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f not found: ID does not exist" containerID="871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f"
Jul 28 17:47:02.787756 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.787673    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f"} err="failed to get container status \"871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f\": rpc error: code = NotFound desc = could not find container \"871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f\": container with ID starting with 871a147cbbd1b19e5ba5a9db39e24e35ae2e4cd10648971d3412ec15903c470f not found: ID does not exist"
Jul 28 17:47:02.790425 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.790406    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"]
Jul 28 17:47:02.795460 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.795441    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29d9e9-7c68bd5d7d-t8pq2"]
Jul 28 17:47:02.809987 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.809969    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"]
Jul 28 17:47:02.816864 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.816847    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29d9e9-844d6db5b4-znqzl"]
Jul 28 17:47:02.829599 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.829580    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"]
Jul 28 17:47:02.836101 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:02.836081    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29d9e9-74bb967f85-5bdhx"]
Jul 28 17:47:03.871551 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:03.871514    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" path="/var/lib/kubelet/pods/7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8/volumes"
Jul 28 17:47:03.871907 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:03.871892    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdbfeab3-497b-4d9a-8729-06746b855fef" path="/var/lib/kubelet/pods/bdbfeab3-497b-4d9a-8729-06746b855fef/volumes"
Jul 28 17:47:03.872249 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:03.872234    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c98aa562-ff57-42fc-a947-f58e906a6f25" path="/var/lib/kubelet/pods/c98aa562-ff57-42fc-a947-f58e906a6f25/volumes"
Jul 28 17:47:18.068741 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.068708    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"]
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069220    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fecb1970-4f1e-437b-9598-dd0c49d6a027" containerName="payload-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069235    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fecb1970-4f1e-437b-9598-dd0c49d6a027" containerName="payload-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069249    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe54260-bcc7-4d12-9cb6-c491894a7d2c" containerName="payload-pre-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069254    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe54260-bcc7-4d12-9cb6-c491894a7d2c" containerName="payload-pre-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069265    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbfeab3-497b-4d9a-8729-06746b855fef" containerName="payload-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069271    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbfeab3-497b-4d9a-8729-06746b855fef" containerName="payload-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069278    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c98aa562-ff57-42fc-a947-f58e906a6f25" containerName="payload-pre-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069283    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c98aa562-ff57-42fc-a947-f58e906a6f25" containerName="payload-pre-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069301    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" containerName="maas-api"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069306    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" containerName="maas-api"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069313    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ca48fc-12c3-4a6f-848d-a08e8d02ed50" containerName="maas-api"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069318    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ca48fc-12c3-4a6f-848d-a08e8d02ed50" containerName="maas-api"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069398    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe54260-bcc7-4d12-9cb6-c491894a7d2c" containerName="payload-pre-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069422    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ca48fc-12c3-4a6f-848d-a08e8d02ed50" containerName="maas-api"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069428    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fecb1970-4f1e-437b-9598-dd0c49d6a027" containerName="payload-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069434    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdbfeab3-497b-4d9a-8729-06746b855fef" containerName="payload-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069440    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c98aa562-ff57-42fc-a947-f58e906a6f25" containerName="payload-pre-processing"
Jul 28 17:47:18.071235 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.069446    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f9bedc6-adc0-4e2c-9f3a-920d68bd65c8" containerName="maas-api"
Jul 28 17:47:18.072287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.072270    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.076377 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.076359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 17:47:18.076697 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.076680    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-f0230c-serving-cert\""
Jul 28 17:47:18.076771 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.076715    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 17:47:18.077234 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.077221    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\""
Jul 28 17:47:18.086001 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.085978    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"]
Jul 28 17:47:18.177304 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.177259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.177304 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.177314    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-maas-api-tls\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.177531 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.177394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7lx\" (UniqueName: \"kubernetes.io/projected/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-kube-api-access-jc7lx\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.278381 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.278328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.278562 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.278403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-maas-api-tls\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.278562 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.278432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7lx\" (UniqueName: \"kubernetes.io/projected/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-kube-api-access-jc7lx\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.278970 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.278942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.280900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.280871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-maas-api-tls\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.293947 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.293923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7lx\" (UniqueName: \"kubernetes.io/projected/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-kube-api-access-jc7lx\") pod \"maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.382368 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.382231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:18.532590 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.532560    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"]
Jul 28 17:47:18.536488 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:47:18.536462    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab9d9f2_737d_4daf_b86f_ca2a9b8b4fee.slice/crio-cf45b216222071bb01553a2be22686164c757495410d01cd05645835edcb3347 WatchSource:0}: Error finding container cf45b216222071bb01553a2be22686164c757495410d01cd05645835edcb3347: Status 404 returned error can't find the container with id cf45b216222071bb01553a2be22686164c757495410d01cd05645835edcb3347
Jul 28 17:47:18.843188 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:18.843153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww" event={"ID":"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee","Type":"ContainerStarted","Data":"cf45b216222071bb01553a2be22686164c757495410d01cd05645835edcb3347"}
Jul 28 17:47:19.229362 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.229317    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"]
Jul 28 17:47:19.232763 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.232746    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.235843 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.235800    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-f0230c\""
Jul 28 17:47:19.235947 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.235916    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-f0230c-dockercfg-zmj5r\""
Jul 28 17:47:19.246861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.246832    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"]
Jul 28 17:47:19.254626 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.254582    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"]
Jul 28 17:47:19.258026 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.258004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.270237 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.270210    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"]
Jul 28 17:47:19.286393 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.286359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttszk\" (UniqueName: \"kubernetes.io/projected/a994b4db-6e7e-4064-9ef5-d28fca3c087b-kube-api-access-ttszk\") pod \"payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp\" (UID: \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.286523 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.286413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a994b4db-6e7e-4064-9ef5-d28fca3c087b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp\" (UID: \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.286523 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.286495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvftw\" (UniqueName: \"kubernetes.io/projected/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-kube-api-access-xvftw\") pod \"payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27\" (UID: \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.286621 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.286535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27\" (UID: \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.387957 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.387922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27\" (UID: \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.388146 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.388019    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttszk\" (UniqueName: \"kubernetes.io/projected/a994b4db-6e7e-4064-9ef5-d28fca3c087b-kube-api-access-ttszk\") pod \"payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp\" (UID: \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.388146 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.388043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a994b4db-6e7e-4064-9ef5-d28fca3c087b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp\" (UID: \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.388146 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.388097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvftw\" (UniqueName: \"kubernetes.io/projected/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-kube-api-access-xvftw\") pod \"payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27\" (UID: \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.388751 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.388727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27\" (UID: \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.388843 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.388772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a994b4db-6e7e-4064-9ef5-d28fca3c087b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp\" (UID: \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.399195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.399123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvftw\" (UniqueName: \"kubernetes.io/projected/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-kube-api-access-xvftw\") pod \"payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27\" (UID: \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.399306 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.399272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttszk\" (UniqueName: \"kubernetes.io/projected/a994b4db-6e7e-4064-9ef5-d28fca3c087b-kube-api-access-ttszk\") pod \"payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp\" (UID: \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.543206 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.543166    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.568501 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.568471    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.688321 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.688285    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"]
Jul 28 17:47:19.692687 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:47:19.692656    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda994b4db_6e7e_4064_9ef5_d28fca3c087b.slice/crio-b10c53730cc434af68d1d7ea8c6d9c2ff28ffbabb18c7bbd3ef7dfe0003d0dd9 WatchSource:0}: Error finding container b10c53730cc434af68d1d7ea8c6d9c2ff28ffbabb18c7bbd3ef7dfe0003d0dd9: Status 404 returned error can't find the container with id b10c53730cc434af68d1d7ea8c6d9c2ff28ffbabb18c7bbd3ef7dfe0003d0dd9
Jul 28 17:47:19.723706 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.723629    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"]
Jul 28 17:47:19.729462 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:47:19.729432    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3250c0c_7e7f_4dc1_95e3_2038894ba7ed.slice/crio-1b9d946bfed968bb219e1770177338cfc0e24e0f4bcf50fbd665527818ed9286 WatchSource:0}: Error finding container 1b9d946bfed968bb219e1770177338cfc0e24e0f4bcf50fbd665527818ed9286: Status 404 returned error can't find the container with id 1b9d946bfed968bb219e1770177338cfc0e24e0f4bcf50fbd665527818ed9286
Jul 28 17:47:19.849885 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.849854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27" event={"ID":"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed","Type":"ContainerStarted","Data":"5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405"}
Jul 28 17:47:19.850114 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.849911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27" event={"ID":"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed","Type":"ContainerStarted","Data":"1b9d946bfed968bb219e1770177338cfc0e24e0f4bcf50fbd665527818ed9286"}
Jul 28 17:47:19.850114 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.849944    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:19.851579 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.851537    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp" event={"ID":"a994b4db-6e7e-4064-9ef5-d28fca3c087b","Type":"ContainerStarted","Data":"8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7"}
Jul 28 17:47:19.851878 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.851592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp" event={"ID":"a994b4db-6e7e-4064-9ef5-d28fca3c087b","Type":"ContainerStarted","Data":"b10c53730cc434af68d1d7ea8c6d9c2ff28ffbabb18c7bbd3ef7dfe0003d0dd9"}
Jul 28 17:47:19.851878 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.851623    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:19.853080 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.853056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww" event={"ID":"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee","Type":"ContainerStarted","Data":"6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a"}
Jul 28 17:47:19.853185 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.853117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:19.873128 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.873085    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27" podStartSLOduration=0.873070183 podStartE2EDuration="873.070183ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:19.871950418 +0000 UTC m=+2310.544278338" watchObservedRunningTime="2026-07-28 17:47:19.873070183 +0000 UTC m=+2310.545398104"
Jul 28 17:47:19.900622 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.900569    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww" podStartSLOduration=1.298245386 podStartE2EDuration="1.900552838s" totalImagesPullingTime="602.307452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:19.898621191 +0000 UTC m=+2310.570949111" watchObservedRunningTime="2026-07-28 17:47:19.900552838 +0000 UTC m=+2310.572880813"
Jul 28 17:47:19.927030 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:19.926928    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp" podStartSLOduration=0.926911616 podStartE2EDuration="926.911616ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:19.924624902 +0000 UTC m=+2310.596952834" watchObservedRunningTime="2026-07-28 17:47:19.926911616 +0000 UTC m=+2310.599239536"
Jul 28 17:47:25.863279 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:25.863248    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 17:47:26.733425 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.733381    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"]
Jul 28 17:47:26.736959 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.736939    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:26.740602 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.740581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-f24b7c-serving-cert\""
Jul 28 17:47:26.757823 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.757798    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"]
Jul 28 17:47:26.858434 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.858337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwf8\" (UniqueName: \"kubernetes.io/projected/e4de72b9-6469-458d-887a-377aa9734052-kube-api-access-tfwf8\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:26.858643 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.858475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de72b9-6469-458d-887a-377aa9734052-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:26.858716 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.858648    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:26.959377 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.959328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:26.959818 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.959422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwf8\" (UniqueName: \"kubernetes.io/projected/e4de72b9-6469-458d-887a-377aa9734052-kube-api-access-tfwf8\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:26.959818 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.959446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de72b9-6469-458d-887a-377aa9734052-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:26.959818 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:47:26.959480    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-serving-cert: secret "maas-api-e2e-shared-b-f24b7c-serving-cert" not found
Jul 28 17:47:26.959818 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:47:26.959544    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls podName:e4de72b9-6469-458d-887a-377aa9734052 nodeName:}" failed. No retries permitted until 2026-07-28 17:47:27.459524335 +0000 UTC m=+2318.131852239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls") pod "maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb" (UID: "e4de72b9-6469-458d-887a-377aa9734052") : secret "maas-api-e2e-shared-b-f24b7c-serving-cert" not found
Jul 28 17:47:26.960070 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.960051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de72b9-6469-458d-887a-377aa9734052-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:26.982080 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:26.982058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwf8\" (UniqueName: \"kubernetes.io/projected/e4de72b9-6469-458d-887a-377aa9734052-kube-api-access-tfwf8\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:27.465177 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:27.465147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:27.465387 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:47:27.465299    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-serving-cert: secret "maas-api-e2e-shared-b-f24b7c-serving-cert" not found
Jul 28 17:47:27.465438 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:47:27.465389    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls podName:e4de72b9-6469-458d-887a-377aa9734052 nodeName:}" failed. No retries permitted until 2026-07-28 17:47:28.46536669 +0000 UTC m=+2319.137694593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls") pod "maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb" (UID: "e4de72b9-6469-458d-887a-377aa9734052") : secret "maas-api-e2e-shared-b-f24b7c-serving-cert" not found
Jul 28 17:47:28.011860 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.011828    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"]
Jul 28 17:47:28.015488 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.015472    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.018287 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.018267    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-f24b7c-dockercfg-c7xzn\""
Jul 28 17:47:28.025936 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.025919    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-f24b7c\""
Jul 28 17:47:28.046638 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.046609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"]
Jul 28 17:47:28.070178 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.070146    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4\" (UID: \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.070379 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.070236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frnq8\" (UniqueName: \"kubernetes.io/projected/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-kube-api-access-frnq8\") pod \"payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4\" (UID: \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.116622 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.116595    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"]
Jul 28 17:47:28.120246 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.120231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.139472 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.139444    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"]
Jul 28 17:47:28.171571 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.171541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4\" (UID: \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.171713 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.171646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnq8\" (UniqueName: \"kubernetes.io/projected/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-kube-api-access-frnq8\") pod \"payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4\" (UID: \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.172221 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.172200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4\" (UID: \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.183156 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.183134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnq8\" (UniqueName: \"kubernetes.io/projected/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-kube-api-access-frnq8\") pod \"payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4\" (UID: \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.272685 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.272598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f\" (UID: \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.272820 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.272723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64pq8\" (UniqueName: \"kubernetes.io/projected/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-kube-api-access-64pq8\") pod \"payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f\" (UID: \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.326323 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.326291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.374622 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.373858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f\" (UID: \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.374622 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.373998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64pq8\" (UniqueName: \"kubernetes.io/projected/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-kube-api-access-64pq8\") pod \"payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f\" (UID: \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.375025 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.374992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f\" (UID: \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.387804 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.387772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64pq8\" (UniqueName: \"kubernetes.io/projected/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-kube-api-access-64pq8\") pod \"payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f\" (UID: \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.429846 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.429817    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.475310 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.475277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:28.477659 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.477636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls\") pod \"maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:28.496164 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.496137    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"]
Jul 28 17:47:28.497546 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:47:28.497509    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda40cacb_0a9a_4cdb_ac23_a19b27ad3147.slice/crio-13f20cf95db8415ca07bcd1957041cc96afb61d561fffea3b298bbee14979097 WatchSource:0}: Error finding container 13f20cf95db8415ca07bcd1957041cc96afb61d561fffea3b298bbee14979097: Status 404 returned error can't find the container with id 13f20cf95db8415ca07bcd1957041cc96afb61d561fffea3b298bbee14979097
Jul 28 17:47:28.547212 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.547179    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:28.597804 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.597629    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"]
Jul 28 17:47:28.603940 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:47:28.603879    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcf5fe9_a316_4d65_ac22_7fa2fa5eea60.slice/crio-6a938938eeaa12ea3ab8e8f4b282ccbea0011c76630f15cbabf3b62973e2cb25 WatchSource:0}: Error finding container 6a938938eeaa12ea3ab8e8f4b282ccbea0011c76630f15cbabf3b62973e2cb25: Status 404 returned error can't find the container with id 6a938938eeaa12ea3ab8e8f4b282ccbea0011c76630f15cbabf3b62973e2cb25
Jul 28 17:47:28.703427 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.702288    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"]
Jul 28 17:47:28.904988 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.904949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4" event={"ID":"da40cacb-0a9a-4cdb-ac23-a19b27ad3147","Type":"ContainerStarted","Data":"c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069"}
Jul 28 17:47:28.905181 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.905004    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4" event={"ID":"da40cacb-0a9a-4cdb-ac23-a19b27ad3147","Type":"ContainerStarted","Data":"13f20cf95db8415ca07bcd1957041cc96afb61d561fffea3b298bbee14979097"}
Jul 28 17:47:28.905181 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.905162    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:28.906248 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.906224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb" event={"ID":"e4de72b9-6469-458d-887a-377aa9734052","Type":"ContainerStarted","Data":"87d3ac9e86f7d504d5a60ef6d947d9d68d6b5749c8c2c6be6074258d986c6222"}
Jul 28 17:47:28.907601 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.907577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f" event={"ID":"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60","Type":"ContainerStarted","Data":"029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612"}
Jul 28 17:47:28.907689 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.907622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f" event={"ID":"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60","Type":"ContainerStarted","Data":"6a938938eeaa12ea3ab8e8f4b282ccbea0011c76630f15cbabf3b62973e2cb25"}
Jul 28 17:47:28.907689 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.907646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:28.933675 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.933630    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4" podStartSLOduration=1.933617367 podStartE2EDuration="1.933617367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:28.932736038 +0000 UTC m=+2319.605063958" watchObservedRunningTime="2026-07-28 17:47:28.933617367 +0000 UTC m=+2319.605945286"
Jul 28 17:47:28.967954 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:28.967897    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f" podStartSLOduration=0.967880468 podStartE2EDuration="967.880468ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:28.967856468 +0000 UTC m=+2319.640184388" watchObservedRunningTime="2026-07-28 17:47:28.967880468 +0000 UTC m=+2319.640208387"
Jul 28 17:47:29.914120 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:29.914085    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb" event={"ID":"e4de72b9-6469-458d-887a-377aa9734052","Type":"ContainerStarted","Data":"2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f"}
Jul 28 17:47:29.914637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:29.914503    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:29.934782 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:29.934720    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb" podStartSLOduration=3.6057741070000002 podStartE2EDuration="3.934703203s" totalImagesPullingTime="328.929096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:29.933581565 +0000 UTC m=+2320.605909487" watchObservedRunningTime="2026-07-28 17:47:29.934703203 +0000 UTC m=+2320.607031125"
Jul 28 17:47:30.860259 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:30.860225    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 17:47:30.860573 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:30.860556    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 17:47:35.925681 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:35.925653    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 17:47:39.915893 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:39.915862    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 17:47:39.916323 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:39.916084    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 17:47:41.207776 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.207748    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"]
Jul 28 17:47:41.211588 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.211567    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.214885 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.214852    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"kube-root-ca.crt\""
Jul 28 17:47:41.214885 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.214854    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"default-dockercfg-gjbh8\""
Jul 28 17:47:41.215042 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.214896    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"openshift-service-ca.crt\""
Jul 28 17:47:41.215042 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.214852    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"auth-test-model-f0230c-kserve-self-signed-certs\""
Jul 28 17:47:41.222970 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.222948    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"]
Jul 28 17:47:41.401730 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.401645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tls-certs\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.401730 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.401704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tmp-dir\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.401949 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.401767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkz4\" (UniqueName: \"kubernetes.io/projected/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-kube-api-access-nkkz4\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.401949 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.401806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-home\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.401949 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.401868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-dshm\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.401949 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.401940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-model-cache\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.502944 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.502872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-dshm\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.502944 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.502921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-model-cache\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.503133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.502952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tls-certs\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.503133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.502969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tmp-dir\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.503133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.502999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkz4\" (UniqueName: \"kubernetes.io/projected/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-kube-api-access-nkkz4\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.503133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.503020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-home\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.503392 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.503322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-home\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.503392 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.503381    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-model-cache\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.503532 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.503456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tmp-dir\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.505140 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.505116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-dshm\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.505524 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.505503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tls-certs\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.512423 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.512403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkz4\" (UniqueName: \"kubernetes.io/projected/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-kube-api-access-nkkz4\") pod \"auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") " pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.521358 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.521322    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:41.654092 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.654062    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"]
Jul 28 17:47:41.657286 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:47:41.657260    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32e1a135_9b5b_4d69_9bd7_b9c7d488b5d3.slice/crio-0b0ee630586ad751951c0db7baab59fb5d79cf5863d60c5ad6e6a2763dc27bc0 WatchSource:0}: Error finding container 0b0ee630586ad751951c0db7baab59fb5d79cf5863d60c5ad6e6a2763dc27bc0: Status 404 returned error can't find the container with id 0b0ee630586ad751951c0db7baab59fb5d79cf5863d60c5ad6e6a2763dc27bc0
Jul 28 17:47:41.979002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.978968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6" event={"ID":"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3","Type":"ContainerStarted","Data":"0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7"}
Jul 28 17:47:41.979002 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:41.979007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6" event={"ID":"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3","Type":"ContainerStarted","Data":"0b0ee630586ad751951c0db7baab59fb5d79cf5863d60c5ad6e6a2763dc27bc0"}
Jul 28 17:47:42.005958 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:42.005913    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6" podStartSLOduration=1.005898356 podStartE2EDuration="1.005898356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:42.00412196 +0000 UTC m=+2332.676449902" watchObservedRunningTime="2026-07-28 17:47:42.005898356 +0000 UTC m=+2332.678226274"
Jul 28 17:47:51.521736 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:51.521692    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:51.522229 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:51.521759    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:51.534395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:51.534371    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:47:52.043881 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:47:52.043851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:48:49.937996 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:48:49.937968    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:48:49.944569 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:48:49.944548    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:52:50.498715 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:52:50.498679    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"]
Jul 28 17:53:05.518033 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.517987    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6" podUID="32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" containerName="main" containerID="cri-o://0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7" gracePeriod=45
Jul 28 17:53:05.763631 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.763607    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:53:05.832292 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832263    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tmp-dir\") pod \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") "
Jul 28 17:53:05.832495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-home\") pod \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") "
Jul 28 17:53:05.832495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832333    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkkz4\" (UniqueName: \"kubernetes.io/projected/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-kube-api-access-nkkz4\") pod \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") "
Jul 28 17:53:05.832495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832399    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-model-cache\") pod \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") "
Jul 28 17:53:05.832495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832424    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tls-certs\") pod \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") "
Jul 28 17:53:05.832495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832488    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-dshm\") pod \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\" (UID: \"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3\") "
Jul 28 17:53:05.832774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832563    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-home" (OuterVolumeSpecName: "home") pod "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" (UID: "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:05.832774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832573    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" (UID: "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:05.832774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832661    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" (UID: "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:05.832900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832794    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.832900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832812    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.832900 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.832824    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.834679 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.834649    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-kube-api-access-nkkz4" (OuterVolumeSpecName: "kube-api-access-nkkz4") pod "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" (UID: "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3"). InnerVolumeSpecName "kube-api-access-nkkz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:05.834910 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.834888    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-dshm" (OuterVolumeSpecName: "dshm") pod "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" (UID: "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:05.834910 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.834902    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" (UID: "32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:05.934022 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.934002    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkkz4\" (UniqueName: \"kubernetes.io/projected/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-kube-api-access-nkkz4\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.934022 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.934021    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.934160 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:05.934031    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:53:06.566112 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.566080    2560 generic.go:356] "Generic (PLEG): container finished" podID="32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" containerID="0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7" exitCode=0
Jul 28 17:53:06.566557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.566122    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"
Jul 28 17:53:06.566557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.566152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6" event={"ID":"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3","Type":"ContainerDied","Data":"0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7"}
Jul 28 17:53:06.566557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.566183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6" event={"ID":"32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3","Type":"ContainerDied","Data":"0b0ee630586ad751951c0db7baab59fb5d79cf5863d60c5ad6e6a2763dc27bc0"}
Jul 28 17:53:06.566557 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.566199    2560 scope.go:117] "RemoveContainer" containerID="0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7"
Jul 28 17:53:06.574623 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.574592    2560 scope.go:117] "RemoveContainer" containerID="0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7"
Jul 28 17:53:06.574835 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:53:06.574810    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7\": container with ID starting with 0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7 not found: ID does not exist" containerID="0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7"
Jul 28 17:53:06.574897 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.574842    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7"} err="failed to get container status \"0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7\": rpc error: code = NotFound desc = could not find container \"0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7\": container with ID starting with 0532a45a398f1d0054dfa83239754fc8d2ceb07fa02e89c014f06415692e1da7 not found: ID does not exist"
Jul 28 17:53:06.585607 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.585585    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"]
Jul 28 17:53:06.590831 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:06.590812    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/auth-test-model-f0230c-kserve-6bb5b9cdc5-bxxm6"]
Jul 28 17:53:07.872936 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:07.872904    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" path="/var/lib/kubelet/pods/32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3/volumes"
Jul 28 17:53:22.212707 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.212676    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"]
Jul 28 17:53:22.213168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.213073    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" containerName="main"
Jul 28 17:53:22.213168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.213107    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" containerName="main"
Jul 28 17:53:22.213245 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.213192    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e1a135-9b5b-4d69-9bd7-b9c7d488b5d3" containerName="main"
Jul 28 17:53:22.217285 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.217268    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.220526 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.220490    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"sub-test-model-f24b7c-kserve-self-signed-certs\""
Jul 28 17:53:22.220526 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.220502    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"kube-root-ca.crt\""
Jul 28 17:53:22.220702 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.220552    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"openshift-service-ca.crt\""
Jul 28 17:53:22.220702 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.220605    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"default-dockercfg-77rkw\""
Jul 28 17:53:22.232530 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.232509    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"]
Jul 28 17:53:22.291404 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.291374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-dshm\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.291547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.291438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-tmp-dir\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.291547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.291461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgpr\" (UniqueName: \"kubernetes.io/projected/85de19a4-aedb-46f5-9a26-10904a53fdf7-kube-api-access-tsgpr\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.291547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.291483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-model-cache\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.291547 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.291507    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-home\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.291685 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.291582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de19a4-aedb-46f5-9a26-10904a53fdf7-tls-certs\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392263 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de19a4-aedb-46f5-9a26-10904a53fdf7-tls-certs\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392428 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-dshm\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392428 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-tmp-dir\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392428 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgpr\" (UniqueName: \"kubernetes.io/projected/85de19a4-aedb-46f5-9a26-10904a53fdf7-kube-api-access-tsgpr\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392594 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-model-cache\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392594 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-home\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392823 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-tmp-dir\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392901 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-model-cache\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.392901 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.392850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-home\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.394653 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.394632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-dshm\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.394843 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.394830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de19a4-aedb-46f5-9a26-10904a53fdf7-tls-certs\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.400794 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.400773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgpr\" (UniqueName: \"kubernetes.io/projected/85de19a4-aedb-46f5-9a26-10904a53fdf7-kube-api-access-tsgpr\") pod \"sub-test-model-f24b7c-kserve-77b84bd859-f7vkl\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") " pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.526193 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.526118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:22.661469 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:22.661444    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"]
Jul 28 17:53:22.663594 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:53:22.663567    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85de19a4_aedb_46f5_9a26_10904a53fdf7.slice/crio-be5b1bd48bdf9564d076da03bba91dbdfd8ced77094f772b625d129ee226f823 WatchSource:0}: Error finding container be5b1bd48bdf9564d076da03bba91dbdfd8ced77094f772b625d129ee226f823: Status 404 returned error can't find the container with id be5b1bd48bdf9564d076da03bba91dbdfd8ced77094f772b625d129ee226f823
Jul 28 17:53:23.652905 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:23.652876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl" event={"ID":"85de19a4-aedb-46f5-9a26-10904a53fdf7","Type":"ContainerStarted","Data":"d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e"}
Jul 28 17:53:23.652905 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:23.652912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl" event={"ID":"85de19a4-aedb-46f5-9a26-10904a53fdf7","Type":"ContainerStarted","Data":"be5b1bd48bdf9564d076da03bba91dbdfd8ced77094f772b625d129ee226f823"}
Jul 28 17:53:23.674266 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:23.674225    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl" podStartSLOduration=1.6742111899999998 podStartE2EDuration="1.67421119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:23.672455716 +0000 UTC m=+2674.344783636" watchObservedRunningTime="2026-07-28 17:53:23.67421119 +0000 UTC m=+2674.346539109"
Jul 28 17:53:32.527163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:32.527119    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:32.527544 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:32.527185    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:32.539466 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:32.539442    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:32.707240 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:32.707205    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:53:49.968806 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:49.968743    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:53:49.976909 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:53:49.976890    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:54:40.311053 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:40.311021    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"]
Jul 28 17:54:55.329161 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.329125    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl" podUID="85de19a4-aedb-46f5-9a26-10904a53fdf7" containerName="main" containerID="cri-o://d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e" gracePeriod=45
Jul 28 17:54:55.569924 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.569895    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:54:55.637825 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.637755    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-tmp-dir\") pod \"85de19a4-aedb-46f5-9a26-10904a53fdf7\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") "
Jul 28 17:54:55.637973 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.637872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-home\") pod \"85de19a4-aedb-46f5-9a26-10904a53fdf7\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") "
Jul 28 17:54:55.637973 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.637903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-model-cache\") pod \"85de19a4-aedb-46f5-9a26-10904a53fdf7\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") "
Jul 28 17:54:55.637973 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.637944    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de19a4-aedb-46f5-9a26-10904a53fdf7-tls-certs\") pod \"85de19a4-aedb-46f5-9a26-10904a53fdf7\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") "
Jul 28 17:54:55.638135 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.637976    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-dshm\") pod \"85de19a4-aedb-46f5-9a26-10904a53fdf7\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") "
Jul 28 17:54:55.638135 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.638000    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsgpr\" (UniqueName: \"kubernetes.io/projected/85de19a4-aedb-46f5-9a26-10904a53fdf7-kube-api-access-tsgpr\") pod \"85de19a4-aedb-46f5-9a26-10904a53fdf7\" (UID: \"85de19a4-aedb-46f5-9a26-10904a53fdf7\") "
Jul 28 17:54:55.638135 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.638007    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85de19a4-aedb-46f5-9a26-10904a53fdf7" (UID: "85de19a4-aedb-46f5-9a26-10904a53fdf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:55.638265 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.638153    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-model-cache" (OuterVolumeSpecName: "model-cache") pod "85de19a4-aedb-46f5-9a26-10904a53fdf7" (UID: "85de19a4-aedb-46f5-9a26-10904a53fdf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:55.638265 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.638167    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-home" (OuterVolumeSpecName: "home") pod "85de19a4-aedb-46f5-9a26-10904a53fdf7" (UID: "85de19a4-aedb-46f5-9a26-10904a53fdf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:55.638381 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.638316    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:54:55.638381 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.638328    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:54:55.638381 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.638338    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:54:55.640091 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.640061    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85de19a4-aedb-46f5-9a26-10904a53fdf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85de19a4-aedb-46f5-9a26-10904a53fdf7" (UID: "85de19a4-aedb-46f5-9a26-10904a53fdf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:54:55.640195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.640092    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-dshm" (OuterVolumeSpecName: "dshm") pod "85de19a4-aedb-46f5-9a26-10904a53fdf7" (UID: "85de19a4-aedb-46f5-9a26-10904a53fdf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:55.640195 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.640170    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85de19a4-aedb-46f5-9a26-10904a53fdf7-kube-api-access-tsgpr" (OuterVolumeSpecName: "kube-api-access-tsgpr") pod "85de19a4-aedb-46f5-9a26-10904a53fdf7" (UID: "85de19a4-aedb-46f5-9a26-10904a53fdf7"). InnerVolumeSpecName "kube-api-access-tsgpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:54:55.739198 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.739168    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85de19a4-aedb-46f5-9a26-10904a53fdf7-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:54:55.739198 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.739194    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsgpr\" (UniqueName: \"kubernetes.io/projected/85de19a4-aedb-46f5-9a26-10904a53fdf7-kube-api-access-tsgpr\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:54:55.739393 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:55.739206    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de19a4-aedb-46f5-9a26-10904a53fdf7-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:54:56.105068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.105031    2560 generic.go:356] "Generic (PLEG): container finished" podID="85de19a4-aedb-46f5-9a26-10904a53fdf7" containerID="d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e" exitCode=0
Jul 28 17:54:56.105258 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.105073    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"
Jul 28 17:54:56.105258 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.105119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl" event={"ID":"85de19a4-aedb-46f5-9a26-10904a53fdf7","Type":"ContainerDied","Data":"d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e"}
Jul 28 17:54:56.105258 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.105170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl" event={"ID":"85de19a4-aedb-46f5-9a26-10904a53fdf7","Type":"ContainerDied","Data":"be5b1bd48bdf9564d076da03bba91dbdfd8ced77094f772b625d129ee226f823"}
Jul 28 17:54:56.105258 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.105194    2560 scope.go:117] "RemoveContainer" containerID="d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e"
Jul 28 17:54:56.114144 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.114130    2560 scope.go:117] "RemoveContainer" containerID="d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e"
Jul 28 17:54:56.114417 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:54:56.114391    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e\": container with ID starting with d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e not found: ID does not exist" containerID="d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e"
Jul 28 17:54:56.114517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.114416    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e"} err="failed to get container status \"d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e\": rpc error: code = NotFound desc = could not find container \"d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e\": container with ID starting with d601f8528a1588cc087e29e10334f7bbb39b56dc3c5991b0e3ea7e96598ac63e not found: ID does not exist"
Jul 28 17:54:56.122065 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.122044    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"]
Jul 28 17:54:56.128835 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:56.128813    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/sub-test-model-f24b7c-kserve-77b84bd859-f7vkl"]
Jul 28 17:54:57.874032 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:54:57.873996    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85de19a4-aedb-46f5-9a26-10904a53fdf7" path="/var/lib/kubelet/pods/85de19a4-aedb-46f5-9a26-10904a53fdf7/volumes"
Jul 28 17:55:08.233745 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.233710    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"]
Jul 28 17:55:08.234133 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.234124    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85de19a4-aedb-46f5-9a26-10904a53fdf7" containerName="main"
Jul 28 17:55:08.234180 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.234135    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="85de19a4-aedb-46f5-9a26-10904a53fdf7" containerName="main"
Jul 28 17:55:08.234229 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.234218    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="85de19a4-aedb-46f5-9a26-10904a53fdf7" containerName="main"
Jul 28 17:55:08.237337 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.237322    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.242175 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.242149    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"openshift-service-ca.crt\""
Jul 28 17:55:08.242175 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.242149    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"kube-root-ca.crt\""
Jul 28 17:55:08.242364 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.242150    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"rate-test-model-f24b7c-kserve-self-signed-certs\""
Jul 28 17:55:08.242364 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.242150    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"default-dockercfg-77rkw\""
Jul 28 17:55:08.253364 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.253319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-model-cache\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.253524 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.253372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-tmp-dir\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.253524 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.253406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqrr\" (UniqueName: \"kubernetes.io/projected/50de1293-d77e-46fe-acf8-57ab7b957de8-kube-api-access-qfqrr\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.253524 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.253503    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-dshm\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.253626 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.253528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50de1293-d77e-46fe-acf8-57ab7b957de8-tls-certs\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.253626 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.253600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-home\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.258542 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.258516    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"]
Jul 28 17:55:08.354567 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.354524    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-dshm\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.354567 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.354569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50de1293-d77e-46fe-acf8-57ab7b957de8-tls-certs\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.354813 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.354604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-home\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.354813 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.354642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-model-cache\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.354813 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.354659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-tmp-dir\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.354813 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.354674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqrr\" (UniqueName: \"kubernetes.io/projected/50de1293-d77e-46fe-acf8-57ab7b957de8-kube-api-access-qfqrr\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.355071 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.355051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-model-cache\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.355120 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.355082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-home\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.355168 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.355128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-tmp-dir\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.356779 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.356760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-dshm\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.357034 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.357019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50de1293-d77e-46fe-acf8-57ab7b957de8-tls-certs\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.367958 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.367927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqrr\" (UniqueName: \"kubernetes.io/projected/50de1293-d77e-46fe-acf8-57ab7b957de8-kube-api-access-qfqrr\") pod \"rate-test-model-f24b7c-kserve-769cb4b459-7vf42\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") " pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.546862 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.546776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:08.689143 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:08.689112    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"]
Jul 28 17:55:08.692104 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:55:08.692079    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50de1293_d77e_46fe_acf8_57ab7b957de8.slice/crio-ab26bd6072f51cd06163bbab6385212b616d8eab7543a694a4c1c9639dd59d39 WatchSource:0}: Error finding container ab26bd6072f51cd06163bbab6385212b616d8eab7543a694a4c1c9639dd59d39: Status 404 returned error can't find the container with id ab26bd6072f51cd06163bbab6385212b616d8eab7543a694a4c1c9639dd59d39
Jul 28 17:55:09.171657 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:09.171619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42" event={"ID":"50de1293-d77e-46fe-acf8-57ab7b957de8","Type":"ContainerStarted","Data":"369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652"}
Jul 28 17:55:09.171825 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:09.171667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42" event={"ID":"50de1293-d77e-46fe-acf8-57ab7b957de8","Type":"ContainerStarted","Data":"ab26bd6072f51cd06163bbab6385212b616d8eab7543a694a4c1c9639dd59d39"}
Jul 28 17:55:09.197153 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:09.197105    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42" podStartSLOduration=1.197089772 podStartE2EDuration="1.197089772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:09.195009189 +0000 UTC m=+2779.867337120" watchObservedRunningTime="2026-07-28 17:55:09.197089772 +0000 UTC m=+2779.869417691"
Jul 28 17:55:18.547847 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:18.547817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:18.548326 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:18.547869    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:18.559748 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:18.559726    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:55:19.229833 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:55:19.229806    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:56:40.392138 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:40.392102    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"]
Jul 28 17:56:55.640026 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:55.639921    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42" podUID="50de1293-d77e-46fe-acf8-57ab7b957de8" containerName="main" containerID="cri-o://369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652" gracePeriod=45
Jul 28 17:56:55.885378 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:55.885330    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:56:56.032612 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.032543    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfqrr\" (UniqueName: \"kubernetes.io/projected/50de1293-d77e-46fe-acf8-57ab7b957de8-kube-api-access-qfqrr\") pod \"50de1293-d77e-46fe-acf8-57ab7b957de8\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") "
Jul 28 17:56:56.032761 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.032635    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-tmp-dir\") pod \"50de1293-d77e-46fe-acf8-57ab7b957de8\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") "
Jul 28 17:56:56.032761 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.032675    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50de1293-d77e-46fe-acf8-57ab7b957de8-tls-certs\") pod \"50de1293-d77e-46fe-acf8-57ab7b957de8\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") "
Jul 28 17:56:56.032761 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.032699    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-home\") pod \"50de1293-d77e-46fe-acf8-57ab7b957de8\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") "
Jul 28 17:56:56.032761 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.032732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-model-cache\") pod \"50de1293-d77e-46fe-acf8-57ab7b957de8\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") "
Jul 28 17:56:56.032999 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.032792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-dshm\") pod \"50de1293-d77e-46fe-acf8-57ab7b957de8\" (UID: \"50de1293-d77e-46fe-acf8-57ab7b957de8\") "
Jul 28 17:56:56.032999 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.032945    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50de1293-d77e-46fe-acf8-57ab7b957de8" (UID: "50de1293-d77e-46fe-acf8-57ab7b957de8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:56.032999 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.032961    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-home" (OuterVolumeSpecName: "home") pod "50de1293-d77e-46fe-acf8-57ab7b957de8" (UID: "50de1293-d77e-46fe-acf8-57ab7b957de8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:56.033148 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.033021    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-model-cache" (OuterVolumeSpecName: "model-cache") pod "50de1293-d77e-46fe-acf8-57ab7b957de8" (UID: "50de1293-d77e-46fe-acf8-57ab7b957de8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:56.033221 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.033198    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:56:56.033276 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.033223    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:56:56.033276 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.033236    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:56:56.034773 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.034748    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50de1293-d77e-46fe-acf8-57ab7b957de8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50de1293-d77e-46fe-acf8-57ab7b957de8" (UID: "50de1293-d77e-46fe-acf8-57ab7b957de8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:56:56.035062 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.035046    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-dshm" (OuterVolumeSpecName: "dshm") pod "50de1293-d77e-46fe-acf8-57ab7b957de8" (UID: "50de1293-d77e-46fe-acf8-57ab7b957de8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:56.035117 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.035063    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50de1293-d77e-46fe-acf8-57ab7b957de8-kube-api-access-qfqrr" (OuterVolumeSpecName: "kube-api-access-qfqrr") pod "50de1293-d77e-46fe-acf8-57ab7b957de8" (UID: "50de1293-d77e-46fe-acf8-57ab7b957de8"). InnerVolumeSpecName "kube-api-access-qfqrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:56:56.134045 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.134022    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfqrr\" (UniqueName: \"kubernetes.io/projected/50de1293-d77e-46fe-acf8-57ab7b957de8-kube-api-access-qfqrr\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:56:56.134045 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.134042    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50de1293-d77e-46fe-acf8-57ab7b957de8-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:56:56.134045 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.134050    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50de1293-d77e-46fe-acf8-57ab7b957de8-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:56:56.705715 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.705684    2560 generic.go:356] "Generic (PLEG): container finished" podID="50de1293-d77e-46fe-acf8-57ab7b957de8" containerID="369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652" exitCode=0
Jul 28 17:56:56.706183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.705724    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"
Jul 28 17:56:56.706183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.705743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42" event={"ID":"50de1293-d77e-46fe-acf8-57ab7b957de8","Type":"ContainerDied","Data":"369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652"}
Jul 28 17:56:56.706183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.705781    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42" event={"ID":"50de1293-d77e-46fe-acf8-57ab7b957de8","Type":"ContainerDied","Data":"ab26bd6072f51cd06163bbab6385212b616d8eab7543a694a4c1c9639dd59d39"}
Jul 28 17:56:56.706183 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.705805    2560 scope.go:117] "RemoveContainer" containerID="369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652"
Jul 28 17:56:56.715014 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.714996    2560 scope.go:117] "RemoveContainer" containerID="369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652"
Jul 28 17:56:56.715266 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:56:56.715248    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652\": container with ID starting with 369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652 not found: ID does not exist" containerID="369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652"
Jul 28 17:56:56.715331 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.715272    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652"} err="failed to get container status \"369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652\": rpc error: code = NotFound desc = could not find container \"369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652\": container with ID starting with 369fe5f1955fdefb5cbbc365c2e5be5dcb78bae569ee3689a607c4d29eeb3652 not found: ID does not exist"
Jul 28 17:56:56.729693 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.729670    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"]
Jul 28 17:56:56.737666 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.737644    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/rate-test-model-f24b7c-kserve-769cb4b459-7vf42"]
Jul 28 17:56:56.959012 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.958939    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7428-139971483941024"]
Jul 28 17:56:56.959397 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.959381    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50de1293-d77e-46fe-acf8-57ab7b957de8" containerName="main"
Jul 28 17:56:56.959450 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.959398    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="50de1293-d77e-46fe-acf8-57ab7b957de8" containerName="main"
Jul 28 17:56:56.959485 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.959463    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="50de1293-d77e-46fe-acf8-57ab7b957de8" containerName="main"
Jul 28 17:56:56.963966 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.963951    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7428-139971483941024"
Jul 28 17:56:56.966807 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.966776    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cf5rx\""
Jul 28 17:56:56.967850 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:56.967827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7428-139971483941024"]
Jul 28 17:56:57.041102 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:57.041064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchgm\" (UniqueName: \"kubernetes.io/projected/5d4b4bfe-4ad7-4d52-abfa-80787bba16b6-kube-api-access-tchgm\") pod \"test-curl-7428-139971483941024\" (UID: \"5d4b4bfe-4ad7-4d52-abfa-80787bba16b6\") " pod="opendatahub/test-curl-7428-139971483941024"
Jul 28 17:56:57.142230 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:57.142189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchgm\" (UniqueName: \"kubernetes.io/projected/5d4b4bfe-4ad7-4d52-abfa-80787bba16b6-kube-api-access-tchgm\") pod \"test-curl-7428-139971483941024\" (UID: \"5d4b4bfe-4ad7-4d52-abfa-80787bba16b6\") " pod="opendatahub/test-curl-7428-139971483941024"
Jul 28 17:56:57.152080 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:57.152055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchgm\" (UniqueName: \"kubernetes.io/projected/5d4b4bfe-4ad7-4d52-abfa-80787bba16b6-kube-api-access-tchgm\") pod \"test-curl-7428-139971483941024\" (UID: \"5d4b4bfe-4ad7-4d52-abfa-80787bba16b6\") " pod="opendatahub/test-curl-7428-139971483941024"
Jul 28 17:56:57.275101 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:57.275018    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7428-139971483941024"
Jul 28 17:56:57.406033 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:57.406006    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7428-139971483941024"]
Jul 28 17:56:57.408614 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:56:57.408583    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d4b4bfe_4ad7_4d52_abfa_80787bba16b6.slice/crio-470ace53ed6aae1c0141e8fc68ede87f2d00cc9ff69cd128b52b4b437dca3bd0 WatchSource:0}: Error finding container 470ace53ed6aae1c0141e8fc68ede87f2d00cc9ff69cd128b52b4b437dca3bd0: Status 404 returned error can't find the container with id 470ace53ed6aae1c0141e8fc68ede87f2d00cc9ff69cd128b52b4b437dca3bd0
Jul 28 17:56:57.409876 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:57.409859    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:56:57.712706 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:57.712670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7428-139971483941024" event={"ID":"5d4b4bfe-4ad7-4d52-abfa-80787bba16b6","Type":"ContainerStarted","Data":"470ace53ed6aae1c0141e8fc68ede87f2d00cc9ff69cd128b52b4b437dca3bd0"}
Jul 28 17:56:57.874501 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:57.874467    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50de1293-d77e-46fe-acf8-57ab7b957de8" path="/var/lib/kubelet/pods/50de1293-d77e-46fe-acf8-57ab7b957de8/volumes"
Jul 28 17:56:58.720237 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:58.720206    2560 generic.go:356] "Generic (PLEG): container finished" podID="5d4b4bfe-4ad7-4d52-abfa-80787bba16b6" containerID="b34e3c44233a0d2f9dbbfe4aaac267074e10bda71bcb4d4f7c8d9a766c7f55c6" exitCode=0
Jul 28 17:56:58.720627 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:58.720300    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7428-139971483941024" event={"ID":"5d4b4bfe-4ad7-4d52-abfa-80787bba16b6","Type":"ContainerDied","Data":"b34e3c44233a0d2f9dbbfe4aaac267074e10bda71bcb4d4f7c8d9a766c7f55c6"}
Jul 28 17:56:59.859382 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:59.859337    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7428-139971483941024"
Jul 28 17:56:59.910170 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:59.910144    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7428-139971483941024_5d4b4bfe-4ad7-4d52-abfa-80787bba16b6/test-curl-7428-139971483941024/0.log"
Jul 28 17:56:59.940483 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:59.940459    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7428-139971483941024"]
Jul 28 17:56:59.944612 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:59.944594    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7428-139971483941024"]
Jul 28 17:56:59.968048 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:59.968032    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tchgm\" (UniqueName: \"kubernetes.io/projected/5d4b4bfe-4ad7-4d52-abfa-80787bba16b6-kube-api-access-tchgm\") pod \"5d4b4bfe-4ad7-4d52-abfa-80787bba16b6\" (UID: \"5d4b4bfe-4ad7-4d52-abfa-80787bba16b6\") "
Jul 28 17:56:59.970069 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:56:59.970042    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4b4bfe-4ad7-4d52-abfa-80787bba16b6-kube-api-access-tchgm" (OuterVolumeSpecName: "kube-api-access-tchgm") pod "5d4b4bfe-4ad7-4d52-abfa-80787bba16b6" (UID: "5d4b4bfe-4ad7-4d52-abfa-80787bba16b6"). InnerVolumeSpecName "kube-api-access-tchgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:57:00.069251 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:00.069213    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tchgm\" (UniqueName: \"kubernetes.io/projected/5d4b4bfe-4ad7-4d52-abfa-80787bba16b6-kube-api-access-tchgm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:57:00.731519 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:00.731490    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7428-139971483941024"
Jul 28 17:57:00.731758 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:00.731535    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="470ace53ed6aae1c0141e8fc68ede87f2d00cc9ff69cd128b52b4b437dca3bd0"
Jul 28 17:57:00.741408 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:57:00.741380    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7428-139971483941024\" is forbidden: User \"system:node:ip-10-0-129-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-222.ec2.internal' and this object" podUID="5d4b4bfe-4ad7-4d52-abfa-80787bba16b6" pod="opendatahub/test-curl-7428-139971483941024"
Jul 28 17:57:01.873135 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:01.873089    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4b4bfe-4ad7-4d52-abfa-80787bba16b6" path="/var/lib/kubelet/pods/5d4b4bfe-4ad7-4d52-abfa-80787bba16b6/volumes"
Jul 28 17:57:03.984228 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:03.984198    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"]
Jul 28 17:57:03.984662 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:03.984648    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4b4bfe-4ad7-4d52-abfa-80787bba16b6" containerName="test-curl-7428-139971483941024"
Jul 28 17:57:03.984706 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:03.984664    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4b4bfe-4ad7-4d52-abfa-80787bba16b6" containerName="test-curl-7428-139971483941024"
Jul 28 17:57:03.984744 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:03.984729    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d4b4bfe-4ad7-4d52-abfa-80787bba16b6" containerName="test-curl-7428-139971483941024"
Jul 28 17:57:03.987938 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:03.987918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:03.990560 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:03.990527    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-20ce8c-serving-cert\""
Jul 28 17:57:04.000851 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.000826    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"]
Jul 28 17:57:04.104497 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.104453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.104709 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.104508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntq4\" (UniqueName: \"kubernetes.io/projected/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-kube-api-access-4ntq4\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.104709 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.104617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.205818 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.205761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.206020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.205860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.206020 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.205892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntq4\" (UniqueName: \"kubernetes.io/projected/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-kube-api-access-4ntq4\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.206020 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:57:04.206000    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-serving-cert: secret "maas-api-e2e-ipp-a-20ce8c-serving-cert" not found
Jul 28 17:57:04.206199 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:57:04.206062    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls podName:a62cb2ff-3374-49ef-b5e7-3ff43b624a6b nodeName:}" failed. No retries permitted until 2026-07-28 17:57:04.706047204 +0000 UTC m=+2895.378375102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls") pod "maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr" (UID: "a62cb2ff-3374-49ef-b5e7-3ff43b624a6b") : secret "maas-api-e2e-ipp-a-20ce8c-serving-cert" not found
Jul 28 17:57:04.206515 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.206493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.216056 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.216037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntq4\" (UniqueName: \"kubernetes.io/projected/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-kube-api-access-4ntq4\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.711170 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.711134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.713260 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.713237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.818921 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.818889    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"]
Jul 28 17:57:04.822468 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.822452    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:04.825866 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.825845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-20ce8c\""
Jul 28 17:57:04.826021 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.826006    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-20ce8c-dockercfg-bs7w7\""
Jul 28 17:57:04.832401 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.832380    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"]
Jul 28 17:57:04.898476 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.898454    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:04.915767 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.915742    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"]
Jul 28 17:57:04.919402 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.919386    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:04.928882 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:04.928860    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"]
Jul 28 17:57:05.014764 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.014733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fc2b\" (UniqueName: \"kubernetes.io/projected/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-kube-api-access-2fc2b\") pod \"payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g\" (UID: \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:05.015237 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.014790    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q49rv\" (UniqueName: \"kubernetes.io/projected/7d08731f-f523-4172-b538-6bee9742f32e-kube-api-access-q49rv\") pod \"payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5\" (UID: \"7d08731f-f523-4172-b538-6bee9742f32e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:05.015237 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.015075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g\" (UID: \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:05.015407 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.015294    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d08731f-f523-4172-b538-6bee9742f32e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5\" (UID: \"7d08731f-f523-4172-b538-6bee9742f32e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:05.036692 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.036661    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"]
Jul 28 17:57:05.040023 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:57:05.040001    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62cb2ff_3374_49ef_b5e7_3ff43b624a6b.slice/crio-32e404f66dac8714289ebaa0ef88c28cbbbfc666e2a7b67b5baa00c52b3512b6 WatchSource:0}: Error finding container 32e404f66dac8714289ebaa0ef88c28cbbbfc666e2a7b67b5baa00c52b3512b6: Status 404 returned error can't find the container with id 32e404f66dac8714289ebaa0ef88c28cbbbfc666e2a7b67b5baa00c52b3512b6
Jul 28 17:57:05.116324 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.116295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g\" (UID: \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:05.116490 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.116338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d08731f-f523-4172-b538-6bee9742f32e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5\" (UID: \"7d08731f-f523-4172-b538-6bee9742f32e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:05.116490 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.116422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fc2b\" (UniqueName: \"kubernetes.io/projected/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-kube-api-access-2fc2b\") pod \"payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g\" (UID: \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:05.116490 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.116470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q49rv\" (UniqueName: \"kubernetes.io/projected/7d08731f-f523-4172-b538-6bee9742f32e-kube-api-access-q49rv\") pod \"payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5\" (UID: \"7d08731f-f523-4172-b538-6bee9742f32e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:05.117024 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.117006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g\" (UID: \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:05.117068 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.117053    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d08731f-f523-4172-b538-6bee9742f32e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5\" (UID: \"7d08731f-f523-4172-b538-6bee9742f32e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:05.125589 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.125571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q49rv\" (UniqueName: \"kubernetes.io/projected/7d08731f-f523-4172-b538-6bee9742f32e-kube-api-access-q49rv\") pod \"payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5\" (UID: \"7d08731f-f523-4172-b538-6bee9742f32e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:05.125678 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.125660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fc2b\" (UniqueName: \"kubernetes.io/projected/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-kube-api-access-2fc2b\") pod \"payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g\" (UID: \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:05.134369 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.134337    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:05.235376 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.235279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:05.273664 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.273634    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"]
Jul 28 17:57:05.276001 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:57:05.275974    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d251f5b_2f0a_42ec_8c71_827b5cbbe43f.slice/crio-c4fa47e9a443b222fdbea1601ce532fca572fd4b05c64001669ae1e78e67cfd3 WatchSource:0}: Error finding container c4fa47e9a443b222fdbea1601ce532fca572fd4b05c64001669ae1e78e67cfd3: Status 404 returned error can't find the container with id c4fa47e9a443b222fdbea1601ce532fca572fd4b05c64001669ae1e78e67cfd3
Jul 28 17:57:05.381042 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.380984    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"]
Jul 28 17:57:05.383771 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:57:05.383748    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d08731f_f523_4172_b538_6bee9742f32e.slice/crio-2c7b993354d5f518233aa9a19001d391f176182d1d45b2f0188892da848e49de WatchSource:0}: Error finding container 2c7b993354d5f518233aa9a19001d391f176182d1d45b2f0188892da848e49de: Status 404 returned error can't find the container with id 2c7b993354d5f518233aa9a19001d391f176182d1d45b2f0188892da848e49de
Jul 28 17:57:05.763919 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.763882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g" event={"ID":"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f","Type":"ContainerStarted","Data":"2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199"}
Jul 28 17:57:05.764130 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.763947    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:05.764130 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.763971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g" event={"ID":"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f","Type":"ContainerStarted","Data":"c4fa47e9a443b222fdbea1601ce532fca572fd4b05c64001669ae1e78e67cfd3"}
Jul 28 17:57:05.765453 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.765428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5" event={"ID":"7d08731f-f523-4172-b538-6bee9742f32e","Type":"ContainerStarted","Data":"853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc"}
Jul 28 17:57:05.765572 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.765473    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5" event={"ID":"7d08731f-f523-4172-b538-6bee9742f32e","Type":"ContainerStarted","Data":"2c7b993354d5f518233aa9a19001d391f176182d1d45b2f0188892da848e49de"}
Jul 28 17:57:05.765572 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.765501    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:05.766814 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.766789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr" event={"ID":"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b","Type":"ContainerStarted","Data":"112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4"}
Jul 28 17:57:05.766913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.766861    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr" event={"ID":"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b","Type":"ContainerStarted","Data":"32e404f66dac8714289ebaa0ef88c28cbbbfc666e2a7b67b5baa00c52b3512b6"}
Jul 28 17:57:05.766913 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.766908    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:05.783659 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.783618    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g" podStartSLOduration=1.78360466 podStartE2EDuration="1.78360466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:05.782098913 +0000 UTC m=+2896.454426837" watchObservedRunningTime="2026-07-28 17:57:05.78360466 +0000 UTC m=+2896.455932579"
Jul 28 17:57:05.804163 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.804111    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr" podStartSLOduration=2.227170904 podStartE2EDuration="2.804095919s" totalImagesPullingTime="576.925015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:05.802906407 +0000 UTC m=+2896.475234340" watchObservedRunningTime="2026-07-28 17:57:05.804095919 +0000 UTC m=+2896.476423838"
Jul 28 17:57:05.827140 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:05.827089    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5" podStartSLOduration=1.827063235 podStartE2EDuration="1.827063235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:05.82454004 +0000 UTC m=+2896.496867986" watchObservedRunningTime="2026-07-28 17:57:05.827063235 +0000 UTC m=+2896.499391154"
Jul 28 17:57:11.777682 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:11.777654    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 17:57:16.774207 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:16.774179    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 17:57:16.774598 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:16.774249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 17:57:24.402335 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.402297    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"]
Jul 28 17:57:24.407812 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.407793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:24.416271 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.416238    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-857ec3\""
Jul 28 17:57:24.416415 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.416275    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-857ec3-dockercfg-6dsml\""
Jul 28 17:57:24.419499 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.419475    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"]
Jul 28 17:57:24.480649 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.480620    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"]
Jul 28 17:57:24.484406 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.484389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:24.489724 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.489698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbz5s\" (UniqueName: \"kubernetes.io/projected/43487e59-64b6-4a30-b381-51c24fa31880-kube-api-access-fbz5s\") pod \"payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8\" (UID: \"43487e59-64b6-4a30-b381-51c24fa31880\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:24.489837 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.489746    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43487e59-64b6-4a30-b381-51c24fa31880-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8\" (UID: \"43487e59-64b6-4a30-b381-51c24fa31880\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:24.498945 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.498921    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"]
Jul 28 17:57:24.591169 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.591138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbz5s\" (UniqueName: \"kubernetes.io/projected/43487e59-64b6-4a30-b381-51c24fa31880-kube-api-access-fbz5s\") pod \"payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8\" (UID: \"43487e59-64b6-4a30-b381-51c24fa31880\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:24.591330 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.591182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43487e59-64b6-4a30-b381-51c24fa31880-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8\" (UID: \"43487e59-64b6-4a30-b381-51c24fa31880\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:24.591330 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.591279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97skd\" (UniqueName: \"kubernetes.io/projected/1627189b-8f52-4ff3-a42a-084670f9ee9b-kube-api-access-97skd\") pod \"payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t\" (UID: \"1627189b-8f52-4ff3-a42a-084670f9ee9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:24.591436 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.591336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627189b-8f52-4ff3-a42a-084670f9ee9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t\" (UID: \"1627189b-8f52-4ff3-a42a-084670f9ee9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:24.591947 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.591927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43487e59-64b6-4a30-b381-51c24fa31880-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8\" (UID: \"43487e59-64b6-4a30-b381-51c24fa31880\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:24.601784 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.601755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbz5s\" (UniqueName: \"kubernetes.io/projected/43487e59-64b6-4a30-b381-51c24fa31880-kube-api-access-fbz5s\") pod \"payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8\" (UID: \"43487e59-64b6-4a30-b381-51c24fa31880\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:24.692495 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.692414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97skd\" (UniqueName: \"kubernetes.io/projected/1627189b-8f52-4ff3-a42a-084670f9ee9b-kube-api-access-97skd\") pod \"payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t\" (UID: \"1627189b-8f52-4ff3-a42a-084670f9ee9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:24.692645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.692556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627189b-8f52-4ff3-a42a-084670f9ee9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t\" (UID: \"1627189b-8f52-4ff3-a42a-084670f9ee9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:24.693223 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.693205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627189b-8f52-4ff3-a42a-084670f9ee9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t\" (UID: \"1627189b-8f52-4ff3-a42a-084670f9ee9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:24.702478 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.702387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97skd\" (UniqueName: \"kubernetes.io/projected/1627189b-8f52-4ff3-a42a-084670f9ee9b-kube-api-access-97skd\") pod \"payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t\" (UID: \"1627189b-8f52-4ff3-a42a-084670f9ee9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:24.719135 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.719110    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:24.793895 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.793799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:24.866012 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:24.865983    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"]
Jul 28 17:57:24.868853 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:57:24.868826    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43487e59_64b6_4a30_b381_51c24fa31880.slice/crio-78cf0129f73cecb84fd3e6f1fadb20b9dcb6718c4a4442056704c1c9716aded8 WatchSource:0}: Error finding container 78cf0129f73cecb84fd3e6f1fadb20b9dcb6718c4a4442056704c1c9716aded8: Status 404 returned error can't find the container with id 78cf0129f73cecb84fd3e6f1fadb20b9dcb6718c4a4442056704c1c9716aded8
Jul 28 17:57:25.148525 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.148498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"]
Jul 28 17:57:25.150499 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:57:25.150471    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1627189b_8f52_4ff3_a42a_084670f9ee9b.slice/crio-32e2ad7d362bdc771f73d8149a6d9e9f7540d63893a794d00b10865be3775f79 WatchSource:0}: Error finding container 32e2ad7d362bdc771f73d8149a6d9e9f7540d63893a794d00b10865be3775f79: Status 404 returned error can't find the container with id 32e2ad7d362bdc771f73d8149a6d9e9f7540d63893a794d00b10865be3775f79
Jul 28 17:57:25.880188 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.880148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8" event={"ID":"43487e59-64b6-4a30-b381-51c24fa31880","Type":"ContainerStarted","Data":"cb23db02a71ff16683878f0e8aa692c37857c2d70ad34631df53be0f5e18265e"}
Jul 28 17:57:25.880648 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.880216    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:25.880648 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.880241    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8" event={"ID":"43487e59-64b6-4a30-b381-51c24fa31880","Type":"ContainerStarted","Data":"78cf0129f73cecb84fd3e6f1fadb20b9dcb6718c4a4442056704c1c9716aded8"}
Jul 28 17:57:25.881497 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.881471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t" event={"ID":"1627189b-8f52-4ff3-a42a-084670f9ee9b","Type":"ContainerStarted","Data":"88da511daa0e7c75fcc3b848e99ce8029bf61b4f60bd1dbd365c1b82ef843c98"}
Jul 28 17:57:25.881625 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.881512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t" event={"ID":"1627189b-8f52-4ff3-a42a-084670f9ee9b","Type":"ContainerStarted","Data":"32e2ad7d362bdc771f73d8149a6d9e9f7540d63893a794d00b10865be3775f79"}
Jul 28 17:57:25.881625 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.881544    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:25.926209 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.926031    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8" podStartSLOduration=1.926011032 podStartE2EDuration="1.926011032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:25.90083737 +0000 UTC m=+2916.573165289" watchObservedRunningTime="2026-07-28 17:57:25.926011032 +0000 UTC m=+2916.598338956"
Jul 28 17:57:25.927656 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:25.927615    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t" podStartSLOduration=1.9276023960000002 podStartE2EDuration="1.927602396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:25.925223601 +0000 UTC m=+2916.597551526" watchObservedRunningTime="2026-07-28 17:57:25.927602396 +0000 UTC m=+2916.599930318"
Jul 28 17:57:36.889667 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:36.889626    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8" podUID="43487e59-64b6-4a30-b381-51c24fa31880" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.75:9004: connect: connection refused"
Jul 28 17:57:36.890236 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:36.890060    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 17:57:46.888839 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:46.888805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 17:57:54.783721 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:54.783668    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-msj2v_09869388-033a-467f-ab36-284134b764c2/payload-processing/0.log"
Jul 28 17:57:54.913788 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:57:54.913758    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t_1627189b-8f52-4ff3-a42a-084670f9ee9b/payload-processing/0.log"
Jul 28 17:58:02.621580 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.621547    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"]
Jul 28 17:58:02.625189 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.625170    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.628766 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.628746    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-20ce8c\"/\"kube-root-ca.crt\""
Jul 28 17:58:02.629573 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.629561    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-20ce8c\"/\"openshift-service-ca.crt\""
Jul 28 17:58:02.629637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.629578    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-20ce8c\"/\"default-dockercfg-9566z\""
Jul 28 17:58:02.629637 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.629602    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-20ce8c\"/\"ipp-route-20ce8c-kserve-self-signed-certs\""
Jul 28 17:58:02.638089 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.638068    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"]
Jul 28 17:58:02.654067 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.654038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-dshm\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.654206 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.654079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-tmp-dir\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.654206 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.654130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4516d7a6-0698-4d5f-befe-2bfda984e83a-tls-certs\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.654206 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.654173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-home\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.654398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.654221    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-model-cache\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.654398 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.654253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcn65\" (UniqueName: \"kubernetes.io/projected/4516d7a6-0698-4d5f-befe-2bfda984e83a-kube-api-access-qcn65\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.754982 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.754952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4516d7a6-0698-4d5f-befe-2bfda984e83a-tls-certs\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.755160 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.754998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-home\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.755160 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.755041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-model-cache\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.755160 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.755083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcn65\" (UniqueName: \"kubernetes.io/projected/4516d7a6-0698-4d5f-befe-2bfda984e83a-kube-api-access-qcn65\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.755318 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.755165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-dshm\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.755318 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.755208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-tmp-dir\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.755588 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.755570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-home\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.755663 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.755566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-model-cache\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.755663 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.755611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-tmp-dir\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.757484 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.757458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-dshm\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.757630 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.757610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4516d7a6-0698-4d5f-befe-2bfda984e83a-tls-certs\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.764995 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.764976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcn65\" (UniqueName: \"kubernetes.io/projected/4516d7a6-0698-4d5f-befe-2bfda984e83a-kube-api-access-qcn65\") pod \"ipp-route-20ce8c-kserve-bdbccf6f5-njkmm\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") " pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:02.934096 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:02.934004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:03.075645 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:03.075616    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"]
Jul 28 17:58:03.077703 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:58:03.077679    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4516d7a6_0698_4d5f_befe_2bfda984e83a.slice/crio-034f757b6053c839accf8795b650c0e101d0896b983f0685b0269279918e62ea WatchSource:0}: Error finding container 034f757b6053c839accf8795b650c0e101d0896b983f0685b0269279918e62ea: Status 404 returned error can't find the container with id 034f757b6053c839accf8795b650c0e101d0896b983f0685b0269279918e62ea
Jul 28 17:58:04.082114 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:04.082079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm" event={"ID":"4516d7a6-0698-4d5f-befe-2bfda984e83a","Type":"ContainerStarted","Data":"e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a"}
Jul 28 17:58:04.082114 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:04.082120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm" event={"ID":"4516d7a6-0698-4d5f-befe-2bfda984e83a","Type":"ContainerStarted","Data":"034f757b6053c839accf8795b650c0e101d0896b983f0685b0269279918e62ea"}
Jul 28 17:58:04.102459 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:04.102412    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm" podStartSLOduration=2.102398103 podStartE2EDuration="2.102398103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:04.100007048 +0000 UTC m=+2954.772334969" watchObservedRunningTime="2026-07-28 17:58:04.102398103 +0000 UTC m=+2954.774726072"
Jul 28 17:58:12.934983 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:12.934887    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:12.935487 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:12.935192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:12.947149 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:12.947121    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:13.138361 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:13.138318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 17:58:50.004307 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:50.004278    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:58:50.013795 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:50.013769    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4c8k5_cb247d8e-6b1f-43c4-9e25-75636655445b/ovn-acl-logging/0.log"
Jul 28 17:58:52.139358 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:52.139326    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5_7d08731f-f523-4172-b538-6bee9742f32e/payload-processing/0.log"
Jul 28 17:58:52.276520 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:52.276494    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-msj2v_09869388-033a-467f-ab36-284134b764c2/payload-processing/0.log"
Jul 28 17:58:52.409624 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:58:52.409555    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t_1627189b-8f52-4ff3-a42a-084670f9ee9b/payload-processing/0.log"
Jul 28 17:59:00.703599 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.703561    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"]
Jul 28 17:59:00.707460 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.707438    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:00.710201 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.710179    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-7fdc90\""
Jul 28 17:59:00.710330 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.710260    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-7fdc90-dockercfg-nflhq\""
Jul 28 17:59:00.720201 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.720178    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"]
Jul 28 17:59:00.762642 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.762592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq\" (UID: \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:00.762764 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.762652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpkrj\" (UniqueName: \"kubernetes.io/projected/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-kube-api-access-dpkrj\") pod \"payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq\" (UID: \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:00.863497 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.863470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpkrj\" (UniqueName: \"kubernetes.io/projected/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-kube-api-access-dpkrj\") pod \"payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq\" (UID: \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:00.863625 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.863606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq\" (UID: \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:00.864274 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.864256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq\" (UID: \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:00.873004 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:00.872977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpkrj\" (UniqueName: \"kubernetes.io/projected/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-kube-api-access-dpkrj\") pod \"payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq\" (UID: \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:01.017794 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:01.017737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:01.153586 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:01.153559    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"]
Jul 28 17:59:01.156271 ip-10-0-129-222 kubenswrapper[2560]: W0728 17:59:01.156242    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d357c0_8f0f_4e8f_bd91_e20a2b849e86.slice/crio-10ef4a5ee2f3c757ec8008389ed4bf08deb8356c279a92e9cbd66446f5e0523b WatchSource:0}: Error finding container 10ef4a5ee2f3c757ec8008389ed4bf08deb8356c279a92e9cbd66446f5e0523b: Status 404 returned error can't find the container with id 10ef4a5ee2f3c757ec8008389ed4bf08deb8356c279a92e9cbd66446f5e0523b
Jul 28 17:59:01.391601 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:01.391570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq" event={"ID":"72d357c0-8f0f-4e8f-bd91-e20a2b849e86","Type":"ContainerStarted","Data":"7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549"}
Jul 28 17:59:01.391774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:01.391618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq" event={"ID":"72d357c0-8f0f-4e8f-bd91-e20a2b849e86","Type":"ContainerStarted","Data":"10ef4a5ee2f3c757ec8008389ed4bf08deb8356c279a92e9cbd66446f5e0523b"}
Jul 28 17:59:01.391774 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:01.391700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:01.409857 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:01.409816    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq" podStartSLOduration=1.409804046 podStartE2EDuration="1.409804046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:01.408765877 +0000 UTC m=+3012.081093796" watchObservedRunningTime="2026-07-28 17:59:01.409804046 +0000 UTC m=+3012.082131965"
Jul 28 17:59:12.400118 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:12.400089    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:31.173517 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.173482    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"]
Jul 28 17:59:31.174124 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.173726    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq" podUID="72d357c0-8f0f-4e8f-bd91-e20a2b849e86" containerName="payload-pre-processing" containerID="cri-o://7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549" gracePeriod=30
Jul 28 17:59:31.433915 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.433860    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:31.531486 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.531454    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpkrj\" (UniqueName: \"kubernetes.io/projected/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-kube-api-access-dpkrj\") pod \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\" (UID: \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\") "
Jul 28 17:59:31.531633 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.531550    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-plugins-config-volume\") pod \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\" (UID: \"72d357c0-8f0f-4e8f-bd91-e20a2b849e86\") "
Jul 28 17:59:31.531908 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.531887    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72d357c0-8f0f-4e8f-bd91-e20a2b849e86" (UID: "72d357c0-8f0f-4e8f-bd91-e20a2b849e86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:59:31.533512 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.533490    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-kube-api-access-dpkrj" (OuterVolumeSpecName: "kube-api-access-dpkrj") pod "72d357c0-8f0f-4e8f-bd91-e20a2b849e86" (UID: "72d357c0-8f0f-4e8f-bd91-e20a2b849e86"). InnerVolumeSpecName "kube-api-access-dpkrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:31.548240 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.548215    2560 generic.go:356] "Generic (PLEG): container finished" podID="72d357c0-8f0f-4e8f-bd91-e20a2b849e86" containerID="7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549" exitCode=0
Jul 28 17:59:31.548395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.548264    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"
Jul 28 17:59:31.548395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.548274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq" event={"ID":"72d357c0-8f0f-4e8f-bd91-e20a2b849e86","Type":"ContainerDied","Data":"7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549"}
Jul 28 17:59:31.548395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.548313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq" event={"ID":"72d357c0-8f0f-4e8f-bd91-e20a2b849e86","Type":"ContainerDied","Data":"10ef4a5ee2f3c757ec8008389ed4bf08deb8356c279a92e9cbd66446f5e0523b"}
Jul 28 17:59:31.548395 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.548334    2560 scope.go:117] "RemoveContainer" containerID="7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549"
Jul 28 17:59:31.557278 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.557262    2560 scope.go:117] "RemoveContainer" containerID="7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549"
Jul 28 17:59:31.557539 ip-10-0-129-222 kubenswrapper[2560]: E0728 17:59:31.557524    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549\": container with ID starting with 7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549 not found: ID does not exist" containerID="7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549"
Jul 28 17:59:31.557593 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.557545    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549"} err="failed to get container status \"7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549\": rpc error: code = NotFound desc = could not find container \"7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549\": container with ID starting with 7eab10c269b59e1ffbda3554bbb1e8b931aa7751774fe25f5a8dd8af05d12549 not found: ID does not exist"
Jul 28 17:59:31.570094 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.570073    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"]
Jul 28 17:59:31.576861 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.576841    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7fdc90-78699fbdf6-c7brq"]
Jul 28 17:59:31.632953 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.632928    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:59:31.632953 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.632950    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpkrj\" (UniqueName: \"kubernetes.io/projected/72d357c0-8f0f-4e8f-bd91-e20a2b849e86-kube-api-access-dpkrj\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 17:59:31.873302 ip-10-0-129-222 kubenswrapper[2560]: I0728 17:59:31.873269    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72d357c0-8f0f-4e8f-bd91-e20a2b849e86" path="/var/lib/kubelet/pods/72d357c0-8f0f-4e8f-bd91-e20a2b849e86/volumes"
Jul 28 18:00:02.789005 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:02.788975    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"]
Jul 28 18:00:02.789550 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:02.789179    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr" podUID="a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" containerName="maas-api" containerID="cri-o://112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4" gracePeriod=30
Jul 28 18:00:02.891137 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:02.891107    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"]
Jul 28 18:00:02.891383 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:02.891333    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5" podUID="7d08731f-f523-4172-b538-6bee9742f32e" containerName="payload-processing" containerID="cri-o://853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc" gracePeriod=30
Jul 28 18:00:02.903697 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:02.903463    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"]
Jul 28 18:00:02.903697 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:02.903668    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g" podUID="6d251f5b-2f0a-42ec-8c71-827b5cbbe43f" containerName="payload-pre-processing" containerID="cri-o://2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199" gracePeriod=30
Jul 28 18:00:03.076148 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.076122    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 18:00:03.178837 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.178815    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 18:00:03.181952 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.181932    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 18:00:03.199712 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.199690    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls\") pod \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") "
Jul 28 18:00:03.199797 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.199766    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ntq4\" (UniqueName: \"kubernetes.io/projected/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-kube-api-access-4ntq4\") pod \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") "
Jul 28 18:00:03.199843 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.199829    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-openshift-service-ca\") pod \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\" (UID: \"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b\") "
Jul 28 18:00:03.200208 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.200183    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" (UID: "a62cb2ff-3374-49ef-b5e7-3ff43b624a6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:00:03.202137 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.202103    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-kube-api-access-4ntq4" (OuterVolumeSpecName: "kube-api-access-4ntq4") pod "a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" (UID: "a62cb2ff-3374-49ef-b5e7-3ff43b624a6b"). InnerVolumeSpecName "kube-api-access-4ntq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:03.202338 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.202300    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" (UID: "a62cb2ff-3374-49ef-b5e7-3ff43b624a6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:03.301119 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301095    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-plugins-config-volume\") pod \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\" (UID: \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\") "
Jul 28 18:00:03.301258 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301164    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q49rv\" (UniqueName: \"kubernetes.io/projected/7d08731f-f523-4172-b538-6bee9742f32e-kube-api-access-q49rv\") pod \"7d08731f-f523-4172-b538-6bee9742f32e\" (UID: \"7d08731f-f523-4172-b538-6bee9742f32e\") "
Jul 28 18:00:03.301258 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301211    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d08731f-f523-4172-b538-6bee9742f32e-plugins-config-volume\") pod \"7d08731f-f523-4172-b538-6bee9742f32e\" (UID: \"7d08731f-f523-4172-b538-6bee9742f32e\") "
Jul 28 18:00:03.301258 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fc2b\" (UniqueName: \"kubernetes.io/projected/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-kube-api-access-2fc2b\") pod \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\" (UID: \"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f\") "
Jul 28 18:00:03.301530 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301509    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ntq4\" (UniqueName: \"kubernetes.io/projected/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-kube-api-access-4ntq4\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:03.301596 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301525    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d251f5b-2f0a-42ec-8c71-827b5cbbe43f" (UID: "6d251f5b-2f0a-42ec-8c71-827b5cbbe43f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:00:03.301596 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301535    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:03.301596 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301576    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:03.301738 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.301610    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d08731f-f523-4172-b538-6bee9742f32e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d08731f-f523-4172-b538-6bee9742f32e" (UID: "7d08731f-f523-4172-b538-6bee9742f32e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:00:03.303207 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.303185    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-kube-api-access-2fc2b" (OuterVolumeSpecName: "kube-api-access-2fc2b") pod "6d251f5b-2f0a-42ec-8c71-827b5cbbe43f" (UID: "6d251f5b-2f0a-42ec-8c71-827b5cbbe43f"). InnerVolumeSpecName "kube-api-access-2fc2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:03.303306 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.303216    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d08731f-f523-4172-b538-6bee9742f32e-kube-api-access-q49rv" (OuterVolumeSpecName: "kube-api-access-q49rv") pod "7d08731f-f523-4172-b538-6bee9742f32e" (UID: "7d08731f-f523-4172-b538-6bee9742f32e"). InnerVolumeSpecName "kube-api-access-q49rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:03.402881 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.402859    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q49rv\" (UniqueName: \"kubernetes.io/projected/7d08731f-f523-4172-b538-6bee9742f32e-kube-api-access-q49rv\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:03.402881 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.402878    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d08731f-f523-4172-b538-6bee9742f32e-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:03.403008 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.402889    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fc2b\" (UniqueName: \"kubernetes.io/projected/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-kube-api-access-2fc2b\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:03.403008 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.402898    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:03.711717 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.711633    2560 generic.go:356] "Generic (PLEG): container finished" podID="6d251f5b-2f0a-42ec-8c71-827b5cbbe43f" containerID="2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199" exitCode=0
Jul 28 18:00:03.711717 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.711678    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"
Jul 28 18:00:03.711917 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.711731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g" event={"ID":"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f","Type":"ContainerDied","Data":"2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199"}
Jul 28 18:00:03.711917 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.711790    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g" event={"ID":"6d251f5b-2f0a-42ec-8c71-827b5cbbe43f","Type":"ContainerDied","Data":"c4fa47e9a443b222fdbea1601ce532fca572fd4b05c64001669ae1e78e67cfd3"}
Jul 28 18:00:03.711917 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.711819    2560 scope.go:117] "RemoveContainer" containerID="2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199"
Jul 28 18:00:03.713177 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.713156    2560 generic.go:356] "Generic (PLEG): container finished" podID="7d08731f-f523-4172-b538-6bee9742f32e" containerID="853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc" exitCode=0
Jul 28 18:00:03.713269 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.713192    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"
Jul 28 18:00:03.713269 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.713244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5" event={"ID":"7d08731f-f523-4172-b538-6bee9742f32e","Type":"ContainerDied","Data":"853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc"}
Jul 28 18:00:03.713409 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.713294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5" event={"ID":"7d08731f-f523-4172-b538-6bee9742f32e","Type":"ContainerDied","Data":"2c7b993354d5f518233aa9a19001d391f176182d1d45b2f0188892da848e49de"}
Jul 28 18:00:03.714690 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.714656    2560 generic.go:356] "Generic (PLEG): container finished" podID="a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" containerID="112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4" exitCode=0
Jul 28 18:00:03.714786 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.714700    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"
Jul 28 18:00:03.714786 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.714708    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr" event={"ID":"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b","Type":"ContainerDied","Data":"112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4"}
Jul 28 18:00:03.714786 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.714742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr" event={"ID":"a62cb2ff-3374-49ef-b5e7-3ff43b624a6b","Type":"ContainerDied","Data":"32e404f66dac8714289ebaa0ef88c28cbbbfc666e2a7b67b5baa00c52b3512b6"}
Jul 28 18:00:03.721497 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.721475    2560 scope.go:117] "RemoveContainer" containerID="2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199"
Jul 28 18:00:03.721747 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:00:03.721726    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199\": container with ID starting with 2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199 not found: ID does not exist" containerID="2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199"
Jul 28 18:00:03.721828 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.721757    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199"} err="failed to get container status \"2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199\": rpc error: code = NotFound desc = could not find container \"2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199\": container with ID starting with 2d21e4e8b8efe9bfd6c182014eee1e58cdfbf0668aeec3650cdff049c4978199 not found: ID does not exist"
Jul 28 18:00:03.721828 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.721778    2560 scope.go:117] "RemoveContainer" containerID="853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc"
Jul 28 18:00:03.730089 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.730074    2560 scope.go:117] "RemoveContainer" containerID="853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc"
Jul 28 18:00:03.730334 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:00:03.730316    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc\": container with ID starting with 853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc not found: ID does not exist" containerID="853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc"
Jul 28 18:00:03.730494 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.730370    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc"} err="failed to get container status \"853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc\": rpc error: code = NotFound desc = could not find container \"853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc\": container with ID starting with 853ca87fe7c78811458b3db61797accf8aa2d613e76c381861ba5cb078ca8adc not found: ID does not exist"
Jul 28 18:00:03.730494 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.730388    2560 scope.go:117] "RemoveContainer" containerID="112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4"
Jul 28 18:00:03.737654 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.737543    2560 scope.go:117] "RemoveContainer" containerID="112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4"
Jul 28 18:00:03.738111 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:00:03.737960    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4\": container with ID starting with 112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4 not found: ID does not exist" containerID="112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4"
Jul 28 18:00:03.738111 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.737991    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4"} err="failed to get container status \"112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4\": rpc error: code = NotFound desc = could not find container \"112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4\": container with ID starting with 112a19d9ebb6b3aa3409e4b731a5a3399e3de946b6c38c4fb423ae5d72458fd4 not found: ID does not exist"
Jul 28 18:00:03.740255 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.740239    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"]
Jul 28 18:00:03.746197 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.746180    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20ce8c-79c99ddd-pl55g"]
Jul 28 18:00:03.759498 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.759475    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"]
Jul 28 18:00:03.766834 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.766810    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20ce8c-7c7855c4d4-tbjp5"]
Jul 28 18:00:03.780261 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.780240    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"]
Jul 28 18:00:03.785991 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.785973    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20ce8c-64f577dd4-cttwr"]
Jul 28 18:00:03.875049 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.875019    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d251f5b-2f0a-42ec-8c71-827b5cbbe43f" path="/var/lib/kubelet/pods/6d251f5b-2f0a-42ec-8c71-827b5cbbe43f/volumes"
Jul 28 18:00:03.875596 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.875577    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d08731f-f523-4172-b538-6bee9742f32e" path="/var/lib/kubelet/pods/7d08731f-f523-4172-b538-6bee9742f32e/volumes"
Jul 28 18:00:03.876096 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:03.876078    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" path="/var/lib/kubelet/pods/a62cb2ff-3374-49ef-b5e7-3ff43b624a6b/volumes"
Jul 28 18:00:08.871142 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:08.871107    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"]
Jul 28 18:00:23.889727 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:23.889694    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm" podUID="4516d7a6-0698-4d5f-befe-2bfda984e83a" containerName="main" containerID="cri-o://e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a" gracePeriod=45
Jul 28 18:00:24.133539 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.133517    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 18:00:24.185040 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.184976    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcn65\" (UniqueName: \"kubernetes.io/projected/4516d7a6-0698-4d5f-befe-2bfda984e83a-kube-api-access-qcn65\") pod \"4516d7a6-0698-4d5f-befe-2bfda984e83a\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") "
Jul 28 18:00:24.185040 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185010    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-tmp-dir\") pod \"4516d7a6-0698-4d5f-befe-2bfda984e83a\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") "
Jul 28 18:00:24.185220 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185082    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-model-cache\") pod \"4516d7a6-0698-4d5f-befe-2bfda984e83a\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") "
Jul 28 18:00:24.185220 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185107    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-home\") pod \"4516d7a6-0698-4d5f-befe-2bfda984e83a\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") "
Jul 28 18:00:24.185220 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185131    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-dshm\") pod \"4516d7a6-0698-4d5f-befe-2bfda984e83a\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") "
Jul 28 18:00:24.185220 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185186    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4516d7a6-0698-4d5f-befe-2bfda984e83a-tls-certs\") pod \"4516d7a6-0698-4d5f-befe-2bfda984e83a\" (UID: \"4516d7a6-0698-4d5f-befe-2bfda984e83a\") "
Jul 28 18:00:24.185437 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185321    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4516d7a6-0698-4d5f-befe-2bfda984e83a" (UID: "4516d7a6-0698-4d5f-befe-2bfda984e83a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:24.185437 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4516d7a6-0698-4d5f-befe-2bfda984e83a" (UID: "4516d7a6-0698-4d5f-befe-2bfda984e83a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:24.190397 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185544    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-home" (OuterVolumeSpecName: "home") pod "4516d7a6-0698-4d5f-befe-2bfda984e83a" (UID: "4516d7a6-0698-4d5f-befe-2bfda984e83a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:24.190397 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185952    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-tmp-dir\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:24.190397 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.185976    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-model-cache\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:24.190397 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.187681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4516d7a6-0698-4d5f-befe-2bfda984e83a-kube-api-access-qcn65" (OuterVolumeSpecName: "kube-api-access-qcn65") pod "4516d7a6-0698-4d5f-befe-2bfda984e83a" (UID: "4516d7a6-0698-4d5f-befe-2bfda984e83a"). InnerVolumeSpecName "kube-api-access-qcn65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:24.198435 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.198405    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-dshm" (OuterVolumeSpecName: "dshm") pod "4516d7a6-0698-4d5f-befe-2bfda984e83a" (UID: "4516d7a6-0698-4d5f-befe-2bfda984e83a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:24.199473 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.199444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4516d7a6-0698-4d5f-befe-2bfda984e83a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4516d7a6-0698-4d5f-befe-2bfda984e83a" (UID: "4516d7a6-0698-4d5f-befe-2bfda984e83a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:24.286715 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.286691    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-home\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:24.286715 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.286712    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4516d7a6-0698-4d5f-befe-2bfda984e83a-dshm\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:24.286839 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.286741    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4516d7a6-0698-4d5f-befe-2bfda984e83a-tls-certs\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:24.286839 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.286751    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcn65\" (UniqueName: \"kubernetes.io/projected/4516d7a6-0698-4d5f-befe-2bfda984e83a-kube-api-access-qcn65\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:00:24.826307 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.826273    2560 generic.go:356] "Generic (PLEG): container finished" podID="4516d7a6-0698-4d5f-befe-2bfda984e83a" containerID="e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a" exitCode=0
Jul 28 18:00:24.826502 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.826326    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"
Jul 28 18:00:24.826502 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.826369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm" event={"ID":"4516d7a6-0698-4d5f-befe-2bfda984e83a","Type":"ContainerDied","Data":"e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a"}
Jul 28 18:00:24.826502 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.826415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm" event={"ID":"4516d7a6-0698-4d5f-befe-2bfda984e83a","Type":"ContainerDied","Data":"034f757b6053c839accf8795b650c0e101d0896b983f0685b0269279918e62ea"}
Jul 28 18:00:24.826502 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.826437    2560 scope.go:117] "RemoveContainer" containerID="e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a"
Jul 28 18:00:24.835648 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.835630    2560 scope.go:117] "RemoveContainer" containerID="e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a"
Jul 28 18:00:24.835901 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:00:24.835883    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a\": container with ID starting with e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a not found: ID does not exist" containerID="e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a"
Jul 28 18:00:24.835945 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.835910    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a"} err="failed to get container status \"e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a\": rpc error: code = NotFound desc = could not find container \"e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a\": container with ID starting with e7a6b3f3b0c562daaa6f546244a2c6f8cca6ff033138cfe25943e21a4952b86a not found: ID does not exist"
Jul 28 18:00:24.854328 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.854304    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"]
Jul 28 18:00:24.856488 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:24.856470    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-20ce8c/ipp-route-20ce8c-kserve-bdbccf6f5-njkmm"]
Jul 28 18:00:25.872022 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:00:25.871987    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4516d7a6-0698-4d5f-befe-2bfda984e83a" path="/var/lib/kubelet/pods/4516d7a6-0698-4d5f-befe-2bfda984e83a/volumes"
Jul 28 18:01:05.798388 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:05.798337    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"]
Jul 28 18:01:05.798998 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:05.798570    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t" podUID="1627189b-8f52-4ff3-a42a-084670f9ee9b" containerName="payload-processing" containerID="cri-o://88da511daa0e7c75fcc3b848e99ce8029bf61b4f60bd1dbd365c1b82ef843c98" gracePeriod=30
Jul 28 18:01:05.805901 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:05.805860    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"]
Jul 28 18:01:05.806151 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:05.806106    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8" podUID="43487e59-64b6-4a30-b381-51c24fa31880" containerName="payload-pre-processing" containerID="cri-o://cb23db02a71ff16683878f0e8aa692c37857c2d70ad34631df53be0f5e18265e" gracePeriod=30
Jul 28 18:01:06.038804 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.038773    2560 generic.go:356] "Generic (PLEG): container finished" podID="43487e59-64b6-4a30-b381-51c24fa31880" containerID="cb23db02a71ff16683878f0e8aa692c37857c2d70ad34631df53be0f5e18265e" exitCode=0
Jul 28 18:01:06.038960 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.038835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8" event={"ID":"43487e59-64b6-4a30-b381-51c24fa31880","Type":"ContainerDied","Data":"cb23db02a71ff16683878f0e8aa692c37857c2d70ad34631df53be0f5e18265e"}
Jul 28 18:01:06.040465 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.040442    2560 generic.go:356] "Generic (PLEG): container finished" podID="1627189b-8f52-4ff3-a42a-084670f9ee9b" containerID="88da511daa0e7c75fcc3b848e99ce8029bf61b4f60bd1dbd365c1b82ef843c98" exitCode=0
Jul 28 18:01:06.040583 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.040521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t" event={"ID":"1627189b-8f52-4ff3-a42a-084670f9ee9b","Type":"ContainerDied","Data":"88da511daa0e7c75fcc3b848e99ce8029bf61b4f60bd1dbd365c1b82ef843c98"}
Jul 28 18:01:06.077127 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.077108    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 18:01:06.078458 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.078442    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 18:01:06.265443 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.265412    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbz5s\" (UniqueName: \"kubernetes.io/projected/43487e59-64b6-4a30-b381-51c24fa31880-kube-api-access-fbz5s\") pod \"43487e59-64b6-4a30-b381-51c24fa31880\" (UID: \"43487e59-64b6-4a30-b381-51c24fa31880\") "
Jul 28 18:01:06.265443 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.265443    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43487e59-64b6-4a30-b381-51c24fa31880-plugins-config-volume\") pod \"43487e59-64b6-4a30-b381-51c24fa31880\" (UID: \"43487e59-64b6-4a30-b381-51c24fa31880\") "
Jul 28 18:01:06.265657 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.265493    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97skd\" (UniqueName: \"kubernetes.io/projected/1627189b-8f52-4ff3-a42a-084670f9ee9b-kube-api-access-97skd\") pod \"1627189b-8f52-4ff3-a42a-084670f9ee9b\" (UID: \"1627189b-8f52-4ff3-a42a-084670f9ee9b\") "
Jul 28 18:01:06.265657 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.265516    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627189b-8f52-4ff3-a42a-084670f9ee9b-plugins-config-volume\") pod \"1627189b-8f52-4ff3-a42a-084670f9ee9b\" (UID: \"1627189b-8f52-4ff3-a42a-084670f9ee9b\") "
Jul 28 18:01:06.265911 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.265890    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43487e59-64b6-4a30-b381-51c24fa31880-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43487e59-64b6-4a30-b381-51c24fa31880" (UID: "43487e59-64b6-4a30-b381-51c24fa31880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:06.265975 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.265923    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1627189b-8f52-4ff3-a42a-084670f9ee9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1627189b-8f52-4ff3-a42a-084670f9ee9b" (UID: "1627189b-8f52-4ff3-a42a-084670f9ee9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:06.267723 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.267699    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1627189b-8f52-4ff3-a42a-084670f9ee9b-kube-api-access-97skd" (OuterVolumeSpecName: "kube-api-access-97skd") pod "1627189b-8f52-4ff3-a42a-084670f9ee9b" (UID: "1627189b-8f52-4ff3-a42a-084670f9ee9b"). InnerVolumeSpecName "kube-api-access-97skd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:06.267830 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.267728    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43487e59-64b6-4a30-b381-51c24fa31880-kube-api-access-fbz5s" (OuterVolumeSpecName: "kube-api-access-fbz5s") pod "43487e59-64b6-4a30-b381-51c24fa31880" (UID: "43487e59-64b6-4a30-b381-51c24fa31880"). InnerVolumeSpecName "kube-api-access-fbz5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:06.367090 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.367035    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbz5s\" (UniqueName: \"kubernetes.io/projected/43487e59-64b6-4a30-b381-51c24fa31880-kube-api-access-fbz5s\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:06.367090 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.367056    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43487e59-64b6-4a30-b381-51c24fa31880-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:06.367090 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.367067    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97skd\" (UniqueName: \"kubernetes.io/projected/1627189b-8f52-4ff3-a42a-084670f9ee9b-kube-api-access-97skd\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:06.367090 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:06.367075    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627189b-8f52-4ff3-a42a-084670f9ee9b-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:07.047124 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.047090    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"
Jul 28 18:01:07.047599 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.047094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8" event={"ID":"43487e59-64b6-4a30-b381-51c24fa31880","Type":"ContainerDied","Data":"78cf0129f73cecb84fd3e6f1fadb20b9dcb6718c4a4442056704c1c9716aded8"}
Jul 28 18:01:07.047599 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.047196    2560 scope.go:117] "RemoveContainer" containerID="cb23db02a71ff16683878f0e8aa692c37857c2d70ad34631df53be0f5e18265e"
Jul 28 18:01:07.048502 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.048416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t" event={"ID":"1627189b-8f52-4ff3-a42a-084670f9ee9b","Type":"ContainerDied","Data":"32e2ad7d362bdc771f73d8149a6d9e9f7540d63893a794d00b10865be3775f79"}
Jul 28 18:01:07.048502 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.048451    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"
Jul 28 18:01:07.058076 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.058056    2560 scope.go:117] "RemoveContainer" containerID="88da511daa0e7c75fcc3b848e99ce8029bf61b4f60bd1dbd365c1b82ef843c98"
Jul 28 18:01:07.076947 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.076924    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"]
Jul 28 18:01:07.081956 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.081936    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-857ec3-5fbc5697cb-tkxm8"]
Jul 28 18:01:07.107728 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.107708    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"]
Jul 28 18:01:07.112111 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.112089    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-857ec3-7cb8d56498-shg2t"]
Jul 28 18:01:07.872969 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.872936    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1627189b-8f52-4ff3-a42a-084670f9ee9b" path="/var/lib/kubelet/pods/1627189b-8f52-4ff3-a42a-084670f9ee9b/volumes"
Jul 28 18:01:07.873291 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:07.873279    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43487e59-64b6-4a30-b381-51c24fa31880" path="/var/lib/kubelet/pods/43487e59-64b6-4a30-b381-51c24fa31880/volumes"
Jul 28 18:01:29.217446 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.217328    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"]
Jul 28 18:01:29.217903 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.217605    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww" podUID="3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" containerName="maas-api" containerID="cri-o://6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a" gracePeriod=30
Jul 28 18:01:29.329022 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.328986    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"]
Jul 28 18:01:29.329512 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.329481    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27" podUID="f3250c0c-7e7f-4dc1-95e3-2038894ba7ed" containerName="payload-processing" containerID="cri-o://5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405" gracePeriod=30
Jul 28 18:01:29.350259 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.350231    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"]
Jul 28 18:01:29.350471 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.350445    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp" podUID="a994b4db-6e7e-4064-9ef5-d28fca3c087b" containerName="payload-pre-processing" containerID="cri-o://8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7" gracePeriod=30
Jul 28 18:01:29.419499 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:01:29.419464    2560 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda994b4db_6e7e_4064_9ef5_d28fca3c087b.slice/crio-8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda994b4db_6e7e_4064_9ef5_d28fca3c087b.slice/crio-conmon-8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 18:01:29.492829 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.492803    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 18:01:29.571821 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.571794    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-maas-api-tls\") pod \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") "
Jul 28 18:01:29.571961 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.571852    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc7lx\" (UniqueName: \"kubernetes.io/projected/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-kube-api-access-jc7lx\") pod \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") "
Jul 28 18:01:29.571961 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.571890    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-openshift-service-ca\") pod \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\" (UID: \"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee\") "
Jul 28 18:01:29.572381 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.572275    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" (UID: "3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:29.575102 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.575078    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-kube-api-access-jc7lx" (OuterVolumeSpecName: "kube-api-access-jc7lx") pod "3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" (UID: "3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee"). InnerVolumeSpecName "kube-api-access-jc7lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:29.575192 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.575123    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" (UID: "3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:01:29.604986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.604956    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 18:01:29.614255 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.614232    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 18:01:29.672975 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.672953    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a994b4db-6e7e-4064-9ef5-d28fca3c087b-plugins-config-volume\") pod \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\" (UID: \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\") "
Jul 28 18:01:29.673101 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.673043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvftw\" (UniqueName: \"kubernetes.io/projected/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-kube-api-access-xvftw\") pod \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\" (UID: \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\") "
Jul 28 18:01:29.673101 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.673097    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttszk\" (UniqueName: \"kubernetes.io/projected/a994b4db-6e7e-4064-9ef5-d28fca3c087b-kube-api-access-ttszk\") pod \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\" (UID: \"a994b4db-6e7e-4064-9ef5-d28fca3c087b\") "
Jul 28 18:01:29.673171 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.673118    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-plugins-config-volume\") pod \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\" (UID: \"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed\") "
Jul 28 18:01:29.673399 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.673377    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:29.673399 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.673401    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc7lx\" (UniqueName: \"kubernetes.io/projected/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-kube-api-access-jc7lx\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:29.673565 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.673414    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:29.673565 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.673450    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a994b4db-6e7e-4064-9ef5-d28fca3c087b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a994b4db-6e7e-4064-9ef5-d28fca3c087b" (UID: "a994b4db-6e7e-4064-9ef5-d28fca3c087b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:29.673565 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.673542    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3250c0c-7e7f-4dc1-95e3-2038894ba7ed" (UID: "f3250c0c-7e7f-4dc1-95e3-2038894ba7ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:29.674990 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.674971    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-kube-api-access-xvftw" (OuterVolumeSpecName: "kube-api-access-xvftw") pod "f3250c0c-7e7f-4dc1-95e3-2038894ba7ed" (UID: "f3250c0c-7e7f-4dc1-95e3-2038894ba7ed"). InnerVolumeSpecName "kube-api-access-xvftw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:29.675041 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.675012    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a994b4db-6e7e-4064-9ef5-d28fca3c087b-kube-api-access-ttszk" (OuterVolumeSpecName: "kube-api-access-ttszk") pod "a994b4db-6e7e-4064-9ef5-d28fca3c087b" (UID: "a994b4db-6e7e-4064-9ef5-d28fca3c087b"). InnerVolumeSpecName "kube-api-access-ttszk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:29.774177 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.774105    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvftw\" (UniqueName: \"kubernetes.io/projected/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-kube-api-access-xvftw\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:29.774177 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.774131    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttszk\" (UniqueName: \"kubernetes.io/projected/a994b4db-6e7e-4064-9ef5-d28fca3c087b-kube-api-access-ttszk\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:29.774177 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.774144    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:29.774177 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:29.774156    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a994b4db-6e7e-4064-9ef5-d28fca3c087b-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:30.160810 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.160770    2560 generic.go:356] "Generic (PLEG): container finished" podID="f3250c0c-7e7f-4dc1-95e3-2038894ba7ed" containerID="5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405" exitCode=0
Jul 28 18:01:30.160957 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.160817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27" event={"ID":"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed","Type":"ContainerDied","Data":"5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405"}
Jul 28 18:01:30.160957 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.160830    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"
Jul 28 18:01:30.160957 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.160900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27" event={"ID":"f3250c0c-7e7f-4dc1-95e3-2038894ba7ed","Type":"ContainerDied","Data":"1b9d946bfed968bb219e1770177338cfc0e24e0f4bcf50fbd665527818ed9286"}
Jul 28 18:01:30.160957 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.160924    2560 scope.go:117] "RemoveContainer" containerID="5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405"
Jul 28 18:01:30.162256 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.162141    2560 generic.go:356] "Generic (PLEG): container finished" podID="a994b4db-6e7e-4064-9ef5-d28fca3c087b" containerID="8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7" exitCode=0
Jul 28 18:01:30.162256 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.162172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp" event={"ID":"a994b4db-6e7e-4064-9ef5-d28fca3c087b","Type":"ContainerDied","Data":"8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7"}
Jul 28 18:01:30.162256 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.162203    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"
Jul 28 18:01:30.162256 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.162224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp" event={"ID":"a994b4db-6e7e-4064-9ef5-d28fca3c087b","Type":"ContainerDied","Data":"b10c53730cc434af68d1d7ea8c6d9c2ff28ffbabb18c7bbd3ef7dfe0003d0dd9"}
Jul 28 18:01:30.163746 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.163728    2560 generic.go:356] "Generic (PLEG): container finished" podID="3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" containerID="6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a" exitCode=0
Jul 28 18:01:30.163860 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.163765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww" event={"ID":"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee","Type":"ContainerDied","Data":"6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a"}
Jul 28 18:01:30.163860 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.163787    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"
Jul 28 18:01:30.163860 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.163809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww" event={"ID":"3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee","Type":"ContainerDied","Data":"cf45b216222071bb01553a2be22686164c757495410d01cd05645835edcb3347"}
Jul 28 18:01:30.170443 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.170428    2560 scope.go:117] "RemoveContainer" containerID="5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405"
Jul 28 18:01:30.170742 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:01:30.170725    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405\": container with ID starting with 5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405 not found: ID does not exist" containerID="5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405"
Jul 28 18:01:30.170823 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.170752    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405"} err="failed to get container status \"5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405\": rpc error: code = NotFound desc = could not find container \"5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405\": container with ID starting with 5b7c2ca119d9a1e4130ab7c3e1ee7bf6158f244cd56828f4b938d4901a2c5405 not found: ID does not exist"
Jul 28 18:01:30.170823 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.170773    2560 scope.go:117] "RemoveContainer" containerID="8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7"
Jul 28 18:01:30.178437 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.178421    2560 scope.go:117] "RemoveContainer" containerID="8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7"
Jul 28 18:01:30.178705 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:01:30.178685    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7\": container with ID starting with 8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7 not found: ID does not exist" containerID="8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7"
Jul 28 18:01:30.178777 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.178715    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7"} err="failed to get container status \"8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7\": rpc error: code = NotFound desc = could not find container \"8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7\": container with ID starting with 8a5babb334377d77e3a97c83dff29bf1513dc21a2c4a03734da0ae9b207560e7 not found: ID does not exist"
Jul 28 18:01:30.178777 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.178738    2560 scope.go:117] "RemoveContainer" containerID="6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a"
Jul 28 18:01:30.186080 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.186065    2560 scope.go:117] "RemoveContainer" containerID="6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a"
Jul 28 18:01:30.186304 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:01:30.186291    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a\": container with ID starting with 6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a not found: ID does not exist" containerID="6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a"
Jul 28 18:01:30.186338 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.186309    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a"} err="failed to get container status \"6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a\": rpc error: code = NotFound desc = could not find container \"6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a\": container with ID starting with 6209118eb44f020c5cc72db755039ae657d69cf10bd0fc6fecd573e87dedba0a not found: ID does not exist"
Jul 28 18:01:30.201303 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.201285    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"]
Jul 28 18:01:30.207987 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.207968    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-f0230c-74cd44c46b-rls27"]
Jul 28 18:01:30.226945 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.226900    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"]
Jul 28 18:01:30.228580 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.228560    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f0230c-6645f5b8dd-7spgp"]
Jul 28 18:01:30.250902 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.250880    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"]
Jul 28 18:01:30.265006 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:30.264983    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f0230c-7dcc5c967-76tww"]
Jul 28 18:01:31.873147 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:31.873117    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" path="/var/lib/kubelet/pods/3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee/volumes"
Jul 28 18:01:31.873509 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:31.873475    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a994b4db-6e7e-4064-9ef5-d28fca3c087b" path="/var/lib/kubelet/pods/a994b4db-6e7e-4064-9ef5-d28fca3c087b/volumes"
Jul 28 18:01:31.873788 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:31.873775    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3250c0c-7e7f-4dc1-95e3-2038894ba7ed" path="/var/lib/kubelet/pods/f3250c0c-7e7f-4dc1-95e3-2038894ba7ed/volumes"
Jul 28 18:01:53.208999 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.208968    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"]
Jul 28 18:01:53.209513 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.209181    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb" podUID="e4de72b9-6469-458d-887a-377aa9734052" containerName="maas-api" containerID="cri-o://2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f" gracePeriod=30
Jul 28 18:01:53.291114 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.291085    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"]
Jul 28 18:01:53.291444 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.291386    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f" podUID="1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60" containerName="payload-processing" containerID="cri-o://029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612" gracePeriod=30
Jul 28 18:01:53.300893 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.300868    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"]
Jul 28 18:01:53.301086 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.301063    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4" podUID="da40cacb-0a9a-4cdb-ac23-a19b27ad3147" containerName="payload-pre-processing" containerID="cri-o://c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069" gracePeriod=30
Jul 28 18:01:53.561630 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.561600    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 18:01:53.584860 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.584483    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 18:01:53.596315 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.596295    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 18:01:53.684237 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.684208    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64pq8\" (UniqueName: \"kubernetes.io/projected/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-kube-api-access-64pq8\") pod \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\" (UID: \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\") "
Jul 28 18:01:53.684392 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.684242    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls\") pod \"e4de72b9-6469-458d-887a-377aa9734052\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") "
Jul 28 18:01:53.684392 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.684292    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-plugins-config-volume\") pod \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\" (UID: \"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60\") "
Jul 28 18:01:53.684392 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.684378    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de72b9-6469-458d-887a-377aa9734052-openshift-service-ca\") pod \"e4de72b9-6469-458d-887a-377aa9734052\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") "
Jul 28 18:01:53.684515 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.684440    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfwf8\" (UniqueName: \"kubernetes.io/projected/e4de72b9-6469-458d-887a-377aa9734052-kube-api-access-tfwf8\") pod \"e4de72b9-6469-458d-887a-377aa9734052\" (UID: \"e4de72b9-6469-458d-887a-377aa9734052\") "
Jul 28 18:01:53.684748 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.684721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60" (UID: "1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:53.684867 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.684719    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4de72b9-6469-458d-887a-377aa9734052-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4de72b9-6469-458d-887a-377aa9734052" (UID: "e4de72b9-6469-458d-887a-377aa9734052"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:53.686381 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.686338    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4de72b9-6469-458d-887a-377aa9734052-kube-api-access-tfwf8" (OuterVolumeSpecName: "kube-api-access-tfwf8") pod "e4de72b9-6469-458d-887a-377aa9734052" (UID: "e4de72b9-6469-458d-887a-377aa9734052"). InnerVolumeSpecName "kube-api-access-tfwf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:53.686472 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.686390    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-kube-api-access-64pq8" (OuterVolumeSpecName: "kube-api-access-64pq8") pod "1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60" (UID: "1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60"). InnerVolumeSpecName "kube-api-access-64pq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:53.686511 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.686494    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4de72b9-6469-458d-887a-377aa9734052" (UID: "e4de72b9-6469-458d-887a-377aa9734052"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:01:53.784951 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.784898    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frnq8\" (UniqueName: \"kubernetes.io/projected/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-kube-api-access-frnq8\") pod \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\" (UID: \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\") "
Jul 28 18:01:53.784951 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.784944    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-plugins-config-volume\") pod \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\" (UID: \"da40cacb-0a9a-4cdb-ac23-a19b27ad3147\") "
Jul 28 18:01:53.785147 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.785134    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfwf8\" (UniqueName: \"kubernetes.io/projected/e4de72b9-6469-458d-887a-377aa9734052-kube-api-access-tfwf8\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:53.785193 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.785150    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64pq8\" (UniqueName: \"kubernetes.io/projected/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-kube-api-access-64pq8\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:53.785193 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.785159    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4de72b9-6469-458d-887a-377aa9734052-maas-api-tls\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:53.785193 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.785169    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:53.785193 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.785178    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de72b9-6469-458d-887a-377aa9734052-openshift-service-ca\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:53.785339 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.785253    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da40cacb-0a9a-4cdb-ac23-a19b27ad3147" (UID: "da40cacb-0a9a-4cdb-ac23-a19b27ad3147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:53.786599 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.786559    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-kube-api-access-frnq8" (OuterVolumeSpecName: "kube-api-access-frnq8") pod "da40cacb-0a9a-4cdb-ac23-a19b27ad3147" (UID: "da40cacb-0a9a-4cdb-ac23-a19b27ad3147"). InnerVolumeSpecName "kube-api-access-frnq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:53.886168 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.886142    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frnq8\" (UniqueName: \"kubernetes.io/projected/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-kube-api-access-frnq8\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:53.886168 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:53.886168    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da40cacb-0a9a-4cdb-ac23-a19b27ad3147-plugins-config-volume\") on node \"ip-10-0-129-222.ec2.internal\" DevicePath \"\""
Jul 28 18:01:54.289181 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.289149    2560 generic.go:356] "Generic (PLEG): container finished" podID="da40cacb-0a9a-4cdb-ac23-a19b27ad3147" containerID="c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069" exitCode=0
Jul 28 18:01:54.289589 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.289192    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"
Jul 28 18:01:54.289589 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.289234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4" event={"ID":"da40cacb-0a9a-4cdb-ac23-a19b27ad3147","Type":"ContainerDied","Data":"c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069"}
Jul 28 18:01:54.289589 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.289294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4" event={"ID":"da40cacb-0a9a-4cdb-ac23-a19b27ad3147","Type":"ContainerDied","Data":"13f20cf95db8415ca07bcd1957041cc96afb61d561fffea3b298bbee14979097"}
Jul 28 18:01:54.289589 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.289325    2560 scope.go:117] "RemoveContainer" containerID="c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069"
Jul 28 18:01:54.290660 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.290642    2560 generic.go:356] "Generic (PLEG): container finished" podID="e4de72b9-6469-458d-887a-377aa9734052" containerID="2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f" exitCode=0
Jul 28 18:01:54.290740 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.290708    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"
Jul 28 18:01:54.290802 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.290757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb" event={"ID":"e4de72b9-6469-458d-887a-377aa9734052","Type":"ContainerDied","Data":"2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f"}
Jul 28 18:01:54.290905 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.290884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb" event={"ID":"e4de72b9-6469-458d-887a-377aa9734052","Type":"ContainerDied","Data":"87d3ac9e86f7d504d5a60ef6d947d9d68d6b5749c8c2c6be6074258d986c6222"}
Jul 28 18:01:54.292252 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.292229    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60" containerID="029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612" exitCode=0
Jul 28 18:01:54.292399 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.292268    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"
Jul 28 18:01:54.292399 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.292319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f" event={"ID":"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60","Type":"ContainerDied","Data":"029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612"}
Jul 28 18:01:54.292399 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.292378    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f" event={"ID":"1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60","Type":"ContainerDied","Data":"6a938938eeaa12ea3ab8e8f4b282ccbea0011c76630f15cbabf3b62973e2cb25"}
Jul 28 18:01:54.298607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.298590    2560 scope.go:117] "RemoveContainer" containerID="c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069"
Jul 28 18:01:54.298854 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:01:54.298838    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069\": container with ID starting with c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069 not found: ID does not exist" containerID="c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069"
Jul 28 18:01:54.298896 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.298868    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069"} err="failed to get container status \"c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069\": rpc error: code = NotFound desc = could not find container \"c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069\": container with ID starting with c9e55313cc58d5035eb97904a4dbb68b6f639664dc413b21637a98ae3d985069 not found: ID does not exist"
Jul 28 18:01:54.298896 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.298884    2560 scope.go:117] "RemoveContainer" containerID="2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f"
Jul 28 18:01:54.305961 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.305946    2560 scope.go:117] "RemoveContainer" containerID="2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f"
Jul 28 18:01:54.306187 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:01:54.306169    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f\": container with ID starting with 2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f not found: ID does not exist" containerID="2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f"
Jul 28 18:01:54.306237 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.306194    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f"} err="failed to get container status \"2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f\": rpc error: code = NotFound desc = could not find container \"2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f\": container with ID starting with 2d94b90458889dd389cbd3c52252e184c69e9104c5bb06283f593863ea08c11f not found: ID does not exist"
Jul 28 18:01:54.306237 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.306209    2560 scope.go:117] "RemoveContainer" containerID="029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612"
Jul 28 18:01:54.313410 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.313393    2560 scope.go:117] "RemoveContainer" containerID="029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612"
Jul 28 18:01:54.313664 ip-10-0-129-222 kubenswrapper[2560]: E0728 18:01:54.313646    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612\": container with ID starting with 029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612 not found: ID does not exist" containerID="029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612"
Jul 28 18:01:54.313731 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.313682    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612"} err="failed to get container status \"029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612\": rpc error: code = NotFound desc = could not find container \"029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612\": container with ID starting with 029fffeff3e228492f0cac479c0b36f2c131f9df53e78021de3ddc8e6c653612 not found: ID does not exist"
Jul 28 18:01:54.321455 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.321433    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"]
Jul 28 18:01:54.328867 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.328846    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f24b7c-6f5f6d6ddd-msmb4"]
Jul 28 18:01:54.352320 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.352299    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"]
Jul 28 18:01:54.361537 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.361520    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f24b7c-684c45fc9d-frd7f"]
Jul 28 18:01:54.379074 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.379050    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"]
Jul 28 18:01:54.382263 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:54.382243    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f24b7c-7b6c88f8d4-4hvjb"]
Jul 28 18:01:55.873483 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:55.873449    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60" path="/var/lib/kubelet/pods/1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60/volumes"
Jul 28 18:01:55.873844 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:55.873793    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da40cacb-0a9a-4cdb-ac23-a19b27ad3147" path="/var/lib/kubelet/pods/da40cacb-0a9a-4cdb-ac23-a19b27ad3147/volumes"
Jul 28 18:01:55.874127 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:01:55.874114    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4de72b9-6469-458d-887a-377aa9734052" path="/var/lib/kubelet/pods/e4de72b9-6469-458d-887a-377aa9734052/volumes"
Jul 28 18:02:13.074395 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:13.074366    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh_22627ec9-dc12-4a35-9982-4e0fb1cdc9b5/copy-manifests/0.log"
Jul 28 18:02:13.086605 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:13.086579    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh_22627ec9-dc12-4a35-9982-4e0fb1cdc9b5/manager/0.log"
Jul 28 18:02:15.924565 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:15.924534    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4d4hl_f5283888-8a98-4238-b681-8e5d241e1e75/kuadrant-console-plugin/0.log"
Jul 28 18:02:16.429732 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:16.429704    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m6jh5_59ed6025-d4d5-41cd-b3b0-c2043f31b462/manager/0.log"
Jul 28 18:02:17.071653 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:17.071623    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b678cfbc-f2cz8_7c92759c-0bcf-46ff-8161-74bde87bb80f/kube-auth-proxy/0.log"
Jul 28 18:02:17.581384 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:17.581338    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-msj2v_09869388-033a-467f-ab36-284134b764c2/payload-processing/0.log"
Jul 28 18:02:18.339304 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:18.339279    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zkljt_1570c290-25a7-4e7b-840c-86ea9e54b349/main/0.log"
Jul 28 18:02:18.583380 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:18.583337    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-whst8_d90c89c3-4171-497b-b9f6-1267a3ffb131/main/0.log"
Jul 28 18:02:24.357541 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357508    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f5qzw/must-gather-vptwb"]
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357867    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a994b4db-6e7e-4064-9ef5-d28fca3c087b" containerName="payload-pre-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357879    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a994b4db-6e7e-4064-9ef5-d28fca3c087b" containerName="payload-pre-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357891    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60" containerName="payload-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357897    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60" containerName="payload-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357909    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3250c0c-7e7f-4dc1-95e3-2038894ba7ed" containerName="payload-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357915    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3250c0c-7e7f-4dc1-95e3-2038894ba7ed" containerName="payload-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357923    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4de72b9-6469-458d-887a-377aa9734052" containerName="maas-api"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357929    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4de72b9-6469-458d-887a-377aa9734052" containerName="maas-api"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357936    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" containerName="maas-api"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357941    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" containerName="maas-api"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357948    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1627189b-8f52-4ff3-a42a-084670f9ee9b" containerName="payload-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357954    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1627189b-8f52-4ff3-a42a-084670f9ee9b" containerName="payload-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357960    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d357c0-8f0f-4e8f-bd91-e20a2b849e86" containerName="payload-pre-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357965    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d357c0-8f0f-4e8f-bd91-e20a2b849e86" containerName="payload-pre-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357971    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da40cacb-0a9a-4cdb-ac23-a19b27ad3147" containerName="payload-pre-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357976    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="da40cacb-0a9a-4cdb-ac23-a19b27ad3147" containerName="payload-pre-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357983    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d08731f-f523-4172-b538-6bee9742f32e" containerName="payload-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357988    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d08731f-f523-4172-b538-6bee9742f32e" containerName="payload-processing"
Jul 28 18:02:24.357986 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.357996    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" containerName="maas-api"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358002    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" containerName="maas-api"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358010    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43487e59-64b6-4a30-b381-51c24fa31880" containerName="payload-pre-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358015    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="43487e59-64b6-4a30-b381-51c24fa31880" containerName="payload-pre-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358026    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d251f5b-2f0a-42ec-8c71-827b5cbbe43f" containerName="payload-pre-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358031    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d251f5b-2f0a-42ec-8c71-827b5cbbe43f" containerName="payload-pre-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358038    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4516d7a6-0698-4d5f-befe-2bfda984e83a" containerName="main"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358043    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4516d7a6-0698-4d5f-befe-2bfda984e83a" containerName="main"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358089    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4de72b9-6469-458d-887a-377aa9734052" containerName="maas-api"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358096    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d251f5b-2f0a-42ec-8c71-827b5cbbe43f" containerName="payload-pre-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358102    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ab9d9f2-737d-4daf-b86f-ca2a9b8b4fee" containerName="maas-api"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358107    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1627189b-8f52-4ff3-a42a-084670f9ee9b" containerName="payload-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358113    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="da40cacb-0a9a-4cdb-ac23-a19b27ad3147" containerName="payload-pre-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358119    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d357c0-8f0f-4e8f-bd91-e20a2b849e86" containerName="payload-pre-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358124    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d08731f-f523-4172-b538-6bee9742f32e" containerName="payload-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358129    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62cb2ff-3374-49ef-b5e7-3ff43b624a6b" containerName="maas-api"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358135    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4516d7a6-0698-4d5f-befe-2bfda984e83a" containerName="main"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358141    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a994b4db-6e7e-4064-9ef5-d28fca3c087b" containerName="payload-pre-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358146    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3250c0c-7e7f-4dc1-95e3-2038894ba7ed" containerName="payload-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358152    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dcf5fe9-a316-4d65-ac22-7fa2fa5eea60" containerName="payload-processing"
Jul 28 18:02:24.358607 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.358157    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="43487e59-64b6-4a30-b381-51c24fa31880" containerName="payload-pre-processing"
Jul 28 18:02:24.361295 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.361278    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f5qzw/must-gather-vptwb"
Jul 28 18:02:24.363758 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.363736    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f5qzw\"/\"openshift-service-ca.crt\""
Jul 28 18:02:24.363834 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.363737    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f5qzw\"/\"default-dockercfg-rmzjs\""
Jul 28 18:02:24.363883 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.363749    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f5qzw\"/\"kube-root-ca.crt\""
Jul 28 18:02:24.372690 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.372665    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f5qzw/must-gather-vptwb"]
Jul 28 18:02:24.441891 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.441864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqrx\" (UniqueName: \"kubernetes.io/projected/a07dc09c-8afd-4827-8fdc-056bb4d2761d-kube-api-access-gsqrx\") pod \"must-gather-vptwb\" (UID: \"a07dc09c-8afd-4827-8fdc-056bb4d2761d\") " pod="openshift-must-gather-f5qzw/must-gather-vptwb"
Jul 28 18:02:24.441998 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.441895    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a07dc09c-8afd-4827-8fdc-056bb4d2761d-must-gather-output\") pod \"must-gather-vptwb\" (UID: \"a07dc09c-8afd-4827-8fdc-056bb4d2761d\") " pod="openshift-must-gather-f5qzw/must-gather-vptwb"
Jul 28 18:02:24.542882 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.542858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqrx\" (UniqueName: \"kubernetes.io/projected/a07dc09c-8afd-4827-8fdc-056bb4d2761d-kube-api-access-gsqrx\") pod \"must-gather-vptwb\" (UID: \"a07dc09c-8afd-4827-8fdc-056bb4d2761d\") " pod="openshift-must-gather-f5qzw/must-gather-vptwb"
Jul 28 18:02:24.542994 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.542887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a07dc09c-8afd-4827-8fdc-056bb4d2761d-must-gather-output\") pod \"must-gather-vptwb\" (UID: \"a07dc09c-8afd-4827-8fdc-056bb4d2761d\") " pod="openshift-must-gather-f5qzw/must-gather-vptwb"
Jul 28 18:02:24.543209 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.543194    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a07dc09c-8afd-4827-8fdc-056bb4d2761d-must-gather-output\") pod \"must-gather-vptwb\" (UID: \"a07dc09c-8afd-4827-8fdc-056bb4d2761d\") " pod="openshift-must-gather-f5qzw/must-gather-vptwb"
Jul 28 18:02:24.553160 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.553133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqrx\" (UniqueName: \"kubernetes.io/projected/a07dc09c-8afd-4827-8fdc-056bb4d2761d-kube-api-access-gsqrx\") pod \"must-gather-vptwb\" (UID: \"a07dc09c-8afd-4827-8fdc-056bb4d2761d\") " pod="openshift-must-gather-f5qzw/must-gather-vptwb"
Jul 28 18:02:24.670161 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.670109    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f5qzw/must-gather-vptwb"
Jul 28 18:02:24.803101 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.803025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f5qzw/must-gather-vptwb"]
Jul 28 18:02:24.805300 ip-10-0-129-222 kubenswrapper[2560]: W0728 18:02:24.805271    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07dc09c_8afd_4827_8fdc_056bb4d2761d.slice/crio-7e68764588b8c94909f4ec4d896d9c304e49ca405051cea100f884a99eefc883 WatchSource:0}: Error finding container 7e68764588b8c94909f4ec4d896d9c304e49ca405051cea100f884a99eefc883: Status 404 returned error can't find the container with id 7e68764588b8c94909f4ec4d896d9c304e49ca405051cea100f884a99eefc883
Jul 28 18:02:24.807239 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:24.807224    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:02:25.440936 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:25.440891    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f5qzw/must-gather-vptwb" event={"ID":"a07dc09c-8afd-4827-8fdc-056bb4d2761d","Type":"ContainerStarted","Data":"7e68764588b8c94909f4ec4d896d9c304e49ca405051cea100f884a99eefc883"}
Jul 28 18:02:26.449561 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:26.449527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f5qzw/must-gather-vptwb" event={"ID":"a07dc09c-8afd-4827-8fdc-056bb4d2761d","Type":"ContainerStarted","Data":"6fcaa43efc1d45c0a051e36655d86a6fcd90f3a51930a5d91ede9ab29a4e7ebc"}
Jul 28 18:02:26.450031 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:26.449573    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f5qzw/must-gather-vptwb" event={"ID":"a07dc09c-8afd-4827-8fdc-056bb4d2761d","Type":"ContainerStarted","Data":"c0d3a80962b5f24a83b1850b446bec0a3af1c1e984000621e23cea6a28a7ebb2"}
Jul 28 18:02:27.424457 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:27.424419    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ng8vd_b207869a-8b53-42dd-a318-2645c2d55a3d/global-pull-secret-syncer/0.log"
Jul 28 18:02:27.565668 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:27.565631    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-txtql_5a65b903-f10a-4ecf-abe1-4cbddd056536/konnectivity-agent/0.log"
Jul 28 18:02:27.592535 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:27.592511    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-222.ec2.internal_e46ae8c13e8686631be2da87086a49c5/haproxy/0.log"
Jul 28 18:02:32.867216 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:32.867180    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4d4hl_f5283888-8a98-4238-b681-8e5d241e1e75/kuadrant-console-plugin/0.log"
Jul 28 18:02:33.259385 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:33.259322    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m6jh5_59ed6025-d4d5-41cd-b3b0-c2043f31b462/manager/0.log"
Jul 28 18:02:34.991039 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:34.990953    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdfb810c-c582-485a-ad8c-cc740d556e33/alertmanager/0.log"
Jul 28 18:02:35.053587 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.053557    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdfb810c-c582-485a-ad8c-cc740d556e33/config-reloader/0.log"
Jul 28 18:02:35.109622 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.109571    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdfb810c-c582-485a-ad8c-cc740d556e33/kube-rbac-proxy-web/0.log"
Jul 28 18:02:35.167332 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.167301    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdfb810c-c582-485a-ad8c-cc740d556e33/kube-rbac-proxy/0.log"
Jul 28 18:02:35.223032 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.223001    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdfb810c-c582-485a-ad8c-cc740d556e33/kube-rbac-proxy-metric/0.log"
Jul 28 18:02:35.262364 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.262276    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdfb810c-c582-485a-ad8c-cc740d556e33/prom-label-proxy/0.log"
Jul 28 18:02:35.293519 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.293478    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdfb810c-c582-485a-ad8c-cc740d556e33/init-config-reloader/0.log"
Jul 28 18:02:35.392046 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.392011    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-k4qrf_068170ef-4e9a-4bf3-bac3-466a630b4bb9/kube-state-metrics/0.log"
Jul 28 18:02:35.415239 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.415199    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-k4qrf_068170ef-4e9a-4bf3-bac3-466a630b4bb9/kube-rbac-proxy-main/0.log"
Jul 28 18:02:35.444322 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.444298    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-k4qrf_068170ef-4e9a-4bf3-bac3-466a630b4bb9/kube-rbac-proxy-self/0.log"
Jul 28 18:02:35.476519 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.476492    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84c9684999-thqzp_3e481fb0-2784-4036-a119-42fec2b0625a/metrics-server/0.log"
Jul 28 18:02:35.508366 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.508307    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-2trmg_5d3fb57c-3f51-467b-b726-20341c8d406a/monitoring-plugin/0.log"
Jul 28 18:02:35.706826 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.706791    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vb65p_de5431c3-9478-45f6-8e5f-cb6d3476ebfb/node-exporter/0.log"
Jul 28 18:02:35.733584 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.733544    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vb65p_de5431c3-9478-45f6-8e5f-cb6d3476ebfb/kube-rbac-proxy/0.log"
Jul 28 18:02:35.760673 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.760642    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vb65p_de5431c3-9478-45f6-8e5f-cb6d3476ebfb/init-textfile/0.log"
Jul 28 18:02:35.793012 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.792985    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2xtjr_8f516ae8-d603-4551-ad56-571deee69bcc/kube-rbac-proxy-main/0.log"
Jul 28 18:02:35.825452 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.825323    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2xtjr_8f516ae8-d603-4551-ad56-571deee69bcc/kube-rbac-proxy-self/0.log"
Jul 28 18:02:35.857715 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.857683    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2xtjr_8f516ae8-d603-4551-ad56-571deee69bcc/openshift-state-metrics/0.log"
Jul 28 18:02:35.901268 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.901230    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd043a1e-c78e-4c16-baa2-2a8b3fa734cc/prometheus/0.log"
Jul 28 18:02:35.928931 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.928903    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd043a1e-c78e-4c16-baa2-2a8b3fa734cc/config-reloader/0.log"
Jul 28 18:02:35.958329 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.958224    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd043a1e-c78e-4c16-baa2-2a8b3fa734cc/thanos-sidecar/0.log"
Jul 28 18:02:35.995638 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:35.995606    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd043a1e-c78e-4c16-baa2-2a8b3fa734cc/kube-rbac-proxy-web/0.log"
Jul 28 18:02:36.037285 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.037241    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd043a1e-c78e-4c16-baa2-2a8b3fa734cc/kube-rbac-proxy/0.log"
Jul 28 18:02:36.065184 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.065150    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd043a1e-c78e-4c16-baa2-2a8b3fa734cc/kube-rbac-proxy-thanos/0.log"
Jul 28 18:02:36.092560 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.092526    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fd043a1e-c78e-4c16-baa2-2a8b3fa734cc/init-config-reloader/0.log"
Jul 28 18:02:36.127961 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.127931    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-h2kgl_e73c443e-0048-4555-be22-9a6ecadf5982/prometheus-operator/0.log"
Jul 28 18:02:36.153697 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.153665    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-h2kgl_e73c443e-0048-4555-be22-9a6ecadf5982/kube-rbac-proxy/0.log"
Jul 28 18:02:36.185478 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.185443    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-9hsw6_521ad204-242e-49c2-9c78-8b7d19da3fd8/prometheus-operator-admission-webhook/0.log"
Jul 28 18:02:36.325390 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.325357    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d48785df-kz9ll_7638afb8-032f-4073-a841-d8e3b8a3e739/thanos-query/0.log"
Jul 28 18:02:36.354175 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.354141    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d48785df-kz9ll_7638afb8-032f-4073-a841-d8e3b8a3e739/kube-rbac-proxy-web/0.log"
Jul 28 18:02:36.385558 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.385530    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d48785df-kz9ll_7638afb8-032f-4073-a841-d8e3b8a3e739/kube-rbac-proxy/0.log"
Jul 28 18:02:36.411578 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.411545    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d48785df-kz9ll_7638afb8-032f-4073-a841-d8e3b8a3e739/prom-label-proxy/0.log"
Jul 28 18:02:36.451755 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.451704    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d48785df-kz9ll_7638afb8-032f-4073-a841-d8e3b8a3e739/kube-rbac-proxy-rules/0.log"
Jul 28 18:02:36.481393 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:36.481359    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59d48785df-kz9ll_7638afb8-032f-4073-a841-d8e3b8a3e739/kube-rbac-proxy-metrics/0.log"
Jul 28 18:02:37.252135 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.252070    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f5qzw/must-gather-vptwb" podStartSLOduration=12.411547283 podStartE2EDuration="13.252048582s" totalImagesPullingTime="840.501299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:26.46958115 +0000 UTC m=+3217.141909070" watchObservedRunningTime="2026-07-28 18:02:37.252048582 +0000 UTC m=+3227.924376501"
Jul 28 18:02:37.254800 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.254771    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"]
Jul 28 18:02:37.259184 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.259163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.267902 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.267877    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"]
Jul 28 18:02:37.386275 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.386246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-podres\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.386486 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.386294    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-proc\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.386486 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.386314    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-lib-modules\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.386486 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.386424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-sys\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.386486 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.386452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqft\" (UniqueName: \"kubernetes.io/projected/0809b4f6-1c83-448a-8256-1174f90fa910-kube-api-access-9zqft\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.487714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-podres\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.487800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-proc\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.487825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-lib-modules\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.487880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-sys\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.487908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqft\" (UniqueName: \"kubernetes.io/projected/0809b4f6-1c83-448a-8256-1174f90fa910-kube-api-access-9zqft\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.488503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-podres\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.488576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-proc\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.488678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-lib-modules\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.488752 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.488721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0809b4f6-1c83-448a-8256-1174f90fa910-sys\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.503424 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.503320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqft\" (UniqueName: \"kubernetes.io/projected/0809b4f6-1c83-448a-8256-1174f90fa910-kube-api-access-9zqft\") pod \"perf-node-gather-daemonset-zkqx2\" (UID: \"0809b4f6-1c83-448a-8256-1174f90fa910\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.574468 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.574425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:37.775977 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:37.773713    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"]
Jul 28 18:02:38.539963 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:38.539927    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2" event={"ID":"0809b4f6-1c83-448a-8256-1174f90fa910","Type":"ContainerStarted","Data":"6b7983070b9fb89054eaaf1c20955e61199a82f9c00c421236d148f48287ddbe"}
Jul 28 18:02:38.540386 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:38.540010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2" event={"ID":"0809b4f6-1c83-448a-8256-1174f90fa910","Type":"ContainerStarted","Data":"41950c3d64e129f8603a77994098a6059914f859332b4051fbe52c7d62a93ae4"}
Jul 28 18:02:38.540386 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:38.540038    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:38.568548 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:38.568477    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2" podStartSLOduration=1.568451934 podStartE2EDuration="1.568451934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:38.567108555 +0000 UTC m=+3229.239436478" watchObservedRunningTime="2026-07-28 18:02:38.568451934 +0000 UTC m=+3229.240779854"
Jul 28 18:02:41.035859 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:41.035826    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rpj8m_0c999d6c-61a0-41cd-bdf9-5e11b3397f50/dns/0.log"
Jul 28 18:02:41.069941 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:41.069909    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rpj8m_0c999d6c-61a0-41cd-bdf9-5e11b3397f50/kube-rbac-proxy/0.log"
Jul 28 18:02:41.109030 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:41.108985    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lgzrj_cc7329f7-672b-4a06-81a4-d942f70a354e/dns-node-resolver/0.log"
Jul 28 18:02:41.858240 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:41.858143    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rgp8f_97e85813-529b-4e8a-9c6b-96684a8dac43/node-ca/0.log"
Jul 28 18:02:43.233107 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:43.233080    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b678cfbc-f2cz8_7c92759c-0bcf-46ff-8161-74bde87bb80f/kube-auth-proxy/0.log"
Jul 28 18:02:43.425632 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:43.425595    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-msj2v_09869388-033a-467f-ab36-284134b764c2/payload-processing/0.log"
Jul 28 18:02:44.393926 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:44.393896    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cpgsl_787aaed5-ae9d-4db2-8961-331a362b78ea/serve-healthcheck-canary/0.log"
Jul 28 18:02:44.561504 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:44.561469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-zkqx2"
Jul 28 18:02:45.271567 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:45.271530    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7vsp_25984382-4f6c-4bc6-ac92-8bf23d4c25b3/kube-rbac-proxy/0.log"
Jul 28 18:02:45.371445 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:45.371416    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7vsp_25984382-4f6c-4bc6-ac92-8bf23d4c25b3/exporter/0.log"
Jul 28 18:02:45.454137 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:45.454109    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m7vsp_25984382-4f6c-4bc6-ac92-8bf23d4c25b3/extractor/0.log"
Jul 28 18:02:48.224241 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:48.224163    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b678cfbc-f2cz8_7c92759c-0bcf-46ff-8161-74bde87bb80f/kube-auth-proxy/0.log"
Jul 28 18:02:48.366555 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:48.366520    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-msj2v_09869388-033a-467f-ab36-284134b764c2/payload-processing/0.log"
Jul 28 18:02:48.960699 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:48.960669    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh_22627ec9-dc12-4a35-9982-4e0fb1cdc9b5/manager/0.log"
Jul 28 18:02:48.993677 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:48.993646    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8c7b7dcd-jk9wh_22627ec9-dc12-4a35-9982-4e0fb1cdc9b5/copy-manifests/0.log"
Jul 28 18:02:50.022138 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:50.022108    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b678cfbc-f2cz8_7c92759c-0bcf-46ff-8161-74bde87bb80f/kube-auth-proxy/0.log"
Jul 28 18:02:50.161628 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:50.161549    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-msj2v_09869388-033a-467f-ab36-284134b764c2/payload-processing/0.log"
Jul 28 18:02:50.581827 ip-10-0-129-222 kubenswrapper[2560]: I0728 18:02:50.581796    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5cdc965c4d-rdf6p_91095dca-a019-4981-9e98-0c45a1a9b5b7/manager/0.log"