Aug 01 00:33:52.630126 ip-10-0-141-142 systemd[1]: Starting Kubernetes Kubelet...
Aug 01 00:33:53.034457 ip-10-0-141-142 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 01 00:33:53.034457 ip-10-0-141-142 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 01 00:33:53.034457 ip-10-0-141-142 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 01 00:33:53.034457 ip-10-0-141-142 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 01 00:33:53.034457 ip-10-0-141-142 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 01 00:33:53.035122 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.034532    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 01 00:33:53.038952 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038936    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 01 00:33:53.038952 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038950    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 01 00:33:53.038952 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038953    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038957    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038962    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038966    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038969    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038973    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038976    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038979    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038982    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038985    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038988    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038991    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.038998    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039001    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039004    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039008    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039011    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039014    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039017    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 01 00:33:53.039040 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039020    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039023    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039025    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039028    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039031    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039033    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039036    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039038    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039041    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039043    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039046    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039049    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039052    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039056    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039058    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039061    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039063    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039066    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039068    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 01 00:33:53.039483 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039071    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039073    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039075    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039078    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039080    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039085    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039088    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039091    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039094    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039096    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039099    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039101    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039103    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039106    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039109    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039112    2570 feature_gate.go:328] unrecognized feature gate: Example
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039115    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039117    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039121    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039123    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 01 00:33:53.039968 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039126    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039128    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039131    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039133    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039136    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039138    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039141    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039144    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039146    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039149    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039151    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039154    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039157    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039159    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039162    2570 feature_gate.go:328] unrecognized feature gate: Example2
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039165    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039167    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039170    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039172    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039174    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 01 00:33:53.040451 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039177    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039179    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039182    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039184    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039187    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039189    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039192    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039195    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039197    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039200    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039203    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039205    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039208    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039210    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039213    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039216    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039218    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039221    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039223    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039965    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 01 00:33:53.040944 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039971    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039974    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039977    2570 feature_gate.go:328] unrecognized feature gate: Example
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039980    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039984    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039987    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039990    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039993    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039995    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.039998    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040002    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040007    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040010    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040013    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040015    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040018    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040021    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040023    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040027    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 01 00:33:53.041427 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040030    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040033    2570 feature_gate.go:328] unrecognized feature gate: Example2
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040036    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040039    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040041    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040044    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040047    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040049    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040052    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040054    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040057    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040059    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040062    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040067    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040070    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040073    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040076    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040080    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040084    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 01 00:33:53.041923 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040087    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040090    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040093    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040095    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040098    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040101    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040103    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040106    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040108    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040111    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040114    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040116    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040119    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040122    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040124    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040127    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040130    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040132    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040135    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040137    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 01 00:33:53.042391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040140    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040142    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040145    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040147    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040150    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040153    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040155    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040157    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040160    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040162    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040165    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040167    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040170    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040172    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040175    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040177    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040180    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040182    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040185    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040187    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 01 00:33:53.042899 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040189    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040192    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040194    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040197    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040200    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040203    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040205    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040209    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040211    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040214    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040217    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040219    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040222    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040224    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040227    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040229    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040232    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040234    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040237    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040239    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 01 00:33:53.043392 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040602    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040607    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040610    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040612    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040616    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040619    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040621    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040624    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040627    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040629    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040632    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040635    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040639    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040643    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040646    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040649    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040652    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040655    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040658    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 01 00:33:53.043925 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040661    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040664    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040666    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040669    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040672    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040674    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040677    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040679    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040682    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040684    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040687    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040689    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040692    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040694    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040697    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040699    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040702    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040705    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040707    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040710    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 01 00:33:53.044377 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040713    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040718    2570 feature_gate.go:328] unrecognized feature gate: Example2
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040721    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040723    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040726    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040728    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040731    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040733    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040736    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040739    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040741    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040744    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040746    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040749    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040751    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040754    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040756    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040760    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040763    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 01 00:33:53.044909 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040766    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040769    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040772    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040775    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040778    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040832    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040837    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040841    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040845    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040848    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040850    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040853    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040856    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040873    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040877    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040880    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040883    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040885    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040888    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040891    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 01 00:33:53.045456 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040895    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040897    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040900    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040902    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040905    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040907    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040910    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040912    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040915    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040917    2570 feature_gate.go:328] unrecognized feature gate: Example
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040920    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040922    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040925    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040927    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040930    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040932    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040935    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040937    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040940    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040942    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 01 00:33:53.045991 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.040945    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041009    2570 flags.go:64] FLAG: --address="0.0.0.0"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041016    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041022    2570 flags.go:64] FLAG: --anonymous-auth="true"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041026    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041031    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041036    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041040    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041044    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041047    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041050    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041053    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041058    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041061    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041064    2570 flags.go:64] FLAG: --cgroup-root=""
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041067    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041070    2570 flags.go:64] FLAG: --client-ca-file=""
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041072    2570 flags.go:64] FLAG: --cloud-provider="external"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041075    2570 flags.go:64] FLAG: --cluster-dns="[]"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041079    2570 flags.go:64] FLAG: --cluster-domain=""
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041082    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041085    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041088    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041091    2570 flags.go:64] FLAG: --container-log-max-files="5"
Aug 01 00:33:53.046497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041094    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041097    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041100    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041103    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041106    2570 flags.go:64] FLAG: --contention-profiling="false"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041109    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041112    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041115    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041118    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041122    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041125    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041127    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041130    2570 flags.go:64] FLAG: --enable-load-reader="false"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041133    2570 flags.go:64] FLAG: --enable-server="true"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041138    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041142    2570 flags.go:64] FLAG: --event-burst="100"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041145    2570 flags.go:64] FLAG: --event-qps="50"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041148    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041151    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041154    2570 flags.go:64] FLAG: --eviction-hard=""
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041158    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041161    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041164    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041167    2570 flags.go:64] FLAG: --eviction-soft=""
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041170    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 01 00:33:53.047090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041173    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041176    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041178    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041181    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041184    2570 flags.go:64] FLAG: --fail-swap-on="true"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041187    2570 flags.go:64] FLAG: --feature-gates=""
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041190    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041193    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041196    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041199    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041202    2570 flags.go:64] FLAG: --healthz-port="10248"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041205    2570 flags.go:64] FLAG: --help="false"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041207    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-141-142.ec2.internal"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041210    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041213    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041216    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041219    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041222    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041225    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041228    2570 flags.go:64] FLAG: --image-service-endpoint=""
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041231    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041234    2570 flags.go:64] FLAG: --kube-api-burst="100"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041238    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041241    2570 flags.go:64] FLAG: --kube-api-qps="50"
Aug 01 00:33:53.047741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041243    2570 flags.go:64] FLAG: --kube-reserved=""
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041246    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041249    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041254    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041256    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041259    2570 flags.go:64] FLAG: --lock-file=""
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041262    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041265    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041268    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041273    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041275    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041278    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041281    2570 flags.go:64] FLAG: --logging-format="text"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041283    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041287    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041289    2570 flags.go:64] FLAG: --manifest-url=""
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041292    2570 flags.go:64] FLAG: --manifest-url-header=""
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041296    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041299    2570 flags.go:64] FLAG: --max-open-files="1000000"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041302    2570 flags.go:64] FLAG: --max-pods="110"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041305    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041308    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041311    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041314    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041317    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 01 00:33:53.048369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041319    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041322    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041328    2570 flags.go:64] FLAG: --node-status-max-images="50"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041331    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041335    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041338    2570 flags.go:64] FLAG: --pod-cidr=""
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041341    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041346    2570 flags.go:64] FLAG: --pod-manifest-path=""
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041349    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041352    2570 flags.go:64] FLAG: --pods-per-core="0"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041356    2570 flags.go:64] FLAG: --port="10250"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041359    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041362    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0adb465279f55be37"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041365    2570 flags.go:64] FLAG: --qos-reserved=""
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041368    2570 flags.go:64] FLAG: --read-only-port="10255"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041371    2570 flags.go:64] FLAG: --register-node="true"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041374    2570 flags.go:64] FLAG: --register-with-taints=""
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041377    2570 flags.go:64] FLAG: --registry-burst="10"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041380    2570 flags.go:64] FLAG: --registry-qps="5"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041383    2570 flags.go:64] FLAG: --reserved-cpus=""
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041386    2570 flags.go:64] FLAG: --reserved-memory=""
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041389    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041392    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041395    2570 flags.go:64] FLAG: --rotate-certificates="false"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041398    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 01 00:33:53.048999 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041401    2570 flags.go:64] FLAG: --runonce="false"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041404    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041407    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041409    2570 flags.go:64] FLAG: --seccomp-default="false"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041412    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041415    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041418    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041421    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041424    2570 flags.go:64] FLAG: --storage-driver-password="root"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041427    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041430    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041433    2570 flags.go:64] FLAG: --storage-driver-user="root"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041437    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041440    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041443    2570 flags.go:64] FLAG: --system-cgroups=""
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041446    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041451    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041455    2570 flags.go:64] FLAG: --tls-cert-file=""
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041458    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041461    2570 flags.go:64] FLAG: --tls-min-version=""
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041464    2570 flags.go:64] FLAG: --tls-private-key-file=""
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041467    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041470    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041473    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041475    2570 flags.go:64] FLAG: --v="2"
Aug 01 00:33:53.049609 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041479    2570 flags.go:64] FLAG: --version="false"
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041483    2570 flags.go:64] FLAG: --vmodule=""
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041486    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.041489    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041607    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041611    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041615    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041619    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041621    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041624    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041627    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041630    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041632    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041635    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041637    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041640    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041642    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041645    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041648    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041651    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 01 00:33:53.050254 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041654    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041656    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041659    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041662    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041666    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041669    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041671    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041674    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041677    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041679    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041682    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041684    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041687    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041690    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041692    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041695    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041697    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041700    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041702    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041705    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 01 00:33:53.050747 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041708    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041710    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041713    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041715    2570 feature_gate.go:328] unrecognized feature gate: Example
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041718    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041720    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041724    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041728    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041730    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041733    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041735    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041738    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041741    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041745    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041747    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041750    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041754    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041756    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041759    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041761    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 01 00:33:53.051256 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041764    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041766    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041769    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041771    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041774    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041778    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041782    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041785    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041788    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041791    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041793    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041796    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041799    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041801    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041804    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041807    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041809    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041812    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041814    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 01 00:33:53.051744 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041817    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041819    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041822    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041824    2570 feature_gate.go:328] unrecognized feature gate: Example2
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041827    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041830    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041832    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041835    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041837    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041841    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041844    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041847    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041849    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041852    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041854    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041857    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041872    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041874    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041877    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041879    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 01 00:33:53.052231 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041882    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041884    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041887    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.041889    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.042457    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.049260    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.049277    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.049295    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.049301    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049356    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049360    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049364    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049367    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049369    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049372    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 01 00:33:53.052727 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049375    2570 feature_gate.go:328] unrecognized feature gate: Example
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049377    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049381    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049386    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049388    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049391    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049394    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049397    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049400    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049402    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049405    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049408    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049410    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049413    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049416    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049418    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049421    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049424    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049426    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 01 00:33:53.053107 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049429    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049432    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049434    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049437    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049445    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049448    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049450    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049453    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049455    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049458    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049460    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049463    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049465    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049467    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049471    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049475    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049478    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049480    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049483    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049485    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 01 00:33:53.053581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049488    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049491    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049493    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049496    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049498    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049500    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049503    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049505    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049508    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049511    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049513    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049516    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049518    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049521    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049523    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049526    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049528    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049539    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049543    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049546    2570 feature_gate.go:328] unrecognized feature gate: Example2
Aug 01 00:33:53.054106 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049557    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049560    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049563    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049566    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049569    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049572    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049575    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049578    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049580    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049583    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049586    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049588    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049591    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049593    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049596    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049598    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049601    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049603    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049606    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049608    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 01 00:33:53.054599 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049611    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049614    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049616    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049619    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049622    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049625    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049628    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049631    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049633    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049636    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049644    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049647    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049649    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049652    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 01 00:33:53.055098 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.049657    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049797    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049802    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049806    2570 feature_gate.go:328] unrecognized feature gate: Example2
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049808    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049811    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049814    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049816    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049819    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049822    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049825    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049827    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049830    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049832    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049835    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049837    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049840    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049842    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049844    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049847    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 01 00:33:53.055440 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049849    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049852    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049856    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049874    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049877    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049880    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049882    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049884    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049887    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049896    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049899    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049902    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049904    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049906    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049909    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049912    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049914    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049917    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049920    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 01 00:33:53.055956 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049922    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049927    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049929    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049932    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049934    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049937    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049945    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049947    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049950    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049952    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049955    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049958    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049960    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049963    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049966    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049968    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049971    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049973    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049976    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 01 00:33:53.056721 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049978    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049982    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049985    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049988    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049996    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.049999    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050002    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050005    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050007    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050010    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050012    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050015    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050017    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050020    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050024    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050026    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050029    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050032    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050034    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 01 00:33:53.057208 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050037    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050040    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050043    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050045    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050048    2570 feature_gate.go:328] unrecognized feature gate: Example
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050050    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050054    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050056    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050059    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050062    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050064    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050067    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050070    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050074    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050077    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050079    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050082    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050085    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050093    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050096    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 01 00:33:53.057835 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050098    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 01 00:33:53.058431 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050101    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 01 00:33:53.058431 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:53.050103    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 01 00:33:53.058431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.050108    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 01 00:33:53.058431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.050206    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 01 00:33:53.058431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.052229    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 01 00:33:53.058431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.053471    2570 server.go:1019] "Starting client certificate rotation"
Aug 01 00:33:53.058431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.053578    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 01 00:33:53.058431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.053626    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 01 00:33:53.059082 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.059065    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 01 00:33:53.060173 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.060158    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 01 00:33:53.069013 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.068997    2570 log.go:25] "Validated CRI v1 runtime API"
Aug 01 00:33:53.070987 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.070970    2570 log.go:25] "Validated CRI v1 image API"
Aug 01 00:33:53.073546 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.073532    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 01 00:33:53.076516 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.076499    2570 fs.go:135] Filesystem UUIDs: map[1ba9563e-f42f-4e0a-a541-8f8d0d6ab155:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a99eea5b-fa69-4474-aadb-dba805ad8aed:/dev/nvme0n1p3]
Aug 01 00:33:53.076567 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.076517    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 01 00:33:53.082241 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.082140    2570 manager.go:216] Machine: {Timestamp:2026-08-01 00:33:53.080833992 +0000 UTC m=+0.336834931 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3132755 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec281f60b90f7955f65d00c8dcea40f6 SystemUUID:ec281f60-b90f-7955-f65d-00c8dcea40f6 BootID:edc73846-66e5-4520-af1a-b703f9b81008 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:41:80:4b:c5:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:41:80:4b:c5:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:37:9c:57:e2:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 01 00:33:53.082241 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.082239    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 01 00:33:53.082352 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.082340    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 01 00:33:53.083444 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.083430    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 01 00:33:53.085593 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.085572    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 01 00:33:53.085716 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.085596    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 01 00:33:53.086275 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.086266    2570 topology_manager.go:138] "Creating topology manager with none policy"
Aug 01 00:33:53.086310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.086276    2570 container_manager_linux.go:309] "Creating device plugin manager"
Aug 01 00:33:53.086310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.086284    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 01 00:33:53.086310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.086298    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 01 00:33:53.087046 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.087037    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 01 00:33:53.090189 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090178    2570 state_mem.go:36] "Initialized new in-memory state store"
Aug 01 00:33:53.090298 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090289    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 01 00:33:53.090332 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090319    2570 kubelet.go:486] "Attempting to sync node with API server"
Aug 01 00:33:53.090359 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090333    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 01 00:33:53.090385 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090363    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 01 00:33:53.090385 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090377    2570 kubelet.go:398] "Adding apiserver pod source"
Aug 01 00:33:53.090441 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090396    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 01 00:33:53.090804 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090792    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 01 00:33:53.090845 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.090825    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 01 00:33:53.093010 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.092989    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 01 00:33:53.095374 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.095359    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 01 00:33:53.095445 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.095389    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 01 00:33:53.096319 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096304    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 01 00:33:53.096319 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096320    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096327    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096333    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096341    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096346    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096354    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096359    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096365    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096370    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096378    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 01 00:33:53.096433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.096384    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 01 00:33:53.097058 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.097049    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 01 00:33:53.097058 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.097057    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 01 00:33:53.099705 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.099687    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 01 00:33:53.099994 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.099982    2570 server.go:1268] "Started kubelet"
Aug 01 00:33:53.100061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.100036    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 01 00:33:53.100184 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.100151    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 01 00:33:53.100254 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.100195    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 01 00:33:53.100413 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.100360    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 01 00:33:53.100503 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.100447    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 01 00:33:53.100799 ip-10-0-141-142 systemd[1]: Started Kubernetes Kubelet.
Aug 01 00:33:53.101150 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.101134    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 01 00:33:53.102848 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.102830    2570 server.go:310] "Adding debug handlers to kubelet server"
Aug 01 00:33:53.105471 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.105452    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 01 00:33:53.105841 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.105825    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 01 00:33:53.105927 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.105910    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 01 00:33:53.105983 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.105932    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:53.106452 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106433    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 01 00:33:53.106533 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106455    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 01 00:33:53.106533 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106431    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 01 00:33:53.106533 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106520    2570 factory.go:55] Registering systemd factory
Aug 01 00:33:53.106533 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106534    2570 factory.go:223] Registration of the systemd container factory successfully
Aug 01 00:33:53.106762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106577    2570 reconstruct.go:97] "Volume reconstruction finished"
Aug 01 00:33:53.106762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106585    2570 reconciler.go:29] "Reconciler: start to sync state"
Aug 01 00:33:53.106762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106723    2570 factory.go:153] Registering CRI-O factory
Aug 01 00:33:53.106762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106736    2570 factory.go:223] Registration of the crio container factory successfully
Aug 01 00:33:53.106981 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106805    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 01 00:33:53.106981 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106836    2570 factory.go:103] Registering Raw factory
Aug 01 00:33:53.106981 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.106850    2570 manager.go:1196] Started watching for new ooms in manager
Aug 01 00:33:53.107240 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.107228    2570 manager.go:349] Starting recovery of all containers
Aug 01 00:33:53.108261 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.108230    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 01 00:33:53.108776 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.108755    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 01 00:33:53.109277 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.108292    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-142.ec2.internal.18c78733a1996579  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-142.ec2.internal,UID:ip-10-0-141-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-142.ec2.internal,},FirstTimestamp:2026-08-01 00:33:53.099957625 +0000 UTC m=+0.355958564,LastTimestamp:2026-08-01 00:33:53.099957625 +0000 UTC m=+0.355958564,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-142.ec2.internal,}"
Aug 01 00:33:53.114160 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.113332    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bbv4g"
Aug 01 00:33:53.118907 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.118888    2570 manager.go:354] Recovery completed
Aug 01 00:33:53.120742 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.120724    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bbv4g"
Aug 01 00:33:53.122923 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.122911    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 01 00:33:53.124720 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.124702    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Aug 01 00:33:53.124809 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.124734    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Aug 01 00:33:53.124809 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.124746    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Aug 01 00:33:53.125219 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.125207    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 01 00:33:53.125282 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.125220    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 01 00:33:53.125282 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.125237    2570 state_mem.go:36] "Initialized new in-memory state store"
Aug 01 00:33:53.125358 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.125223    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 01 00:33:53.126063 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.125945    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Aug 01 00:33:53.126063 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.125975    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Aug 01 00:33:53.126063 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.125990    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Aug 01 00:33:53.126276 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.126262    2570 policy_none.go:49] "None policy: Start"
Aug 01 00:33:53.126312 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.126280    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 01 00:33:53.126312 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.126290    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 01 00:33:53.126832 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.126766    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-142.ec2.internal.18c78733a3133d27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-142.ec2.internal,UID:ip-10-0-141-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-142.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-142.ec2.internal,},FirstTimestamp:2026-08-01 00:33:53.124719911 +0000 UTC m=+0.380720850,LastTimestamp:2026-08-01 00:33:53.124719911 +0000 UTC m=+0.380720850,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-142.ec2.internal,}"
Aug 01 00:33:53.127252 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.127241    2570 policy_none.go:47] "Start"
Aug 01 00:33:53.162532 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.162387    2570 manager.go:337] "Starting Device Plugin manager"
Aug 01 00:33:53.162606 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.162562    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 01 00:33:53.162606 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.162589    2570 server.go:87] "Starting device plugin registration server"
Aug 01 00:33:53.162782 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.162776    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 01 00:33:53.162838 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.162788    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 01 00:33:53.162911 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.162897    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 01 00:33:53.162991 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.162969    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 01 00:33:53.162991 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.162982    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 01 00:33:53.163416 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.163394    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 01 00:33:53.163501 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.163444    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:53.232498 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.232456    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 01 00:33:53.233573 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.233559    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 01 00:33:53.233629 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.233593    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 01 00:33:53.233669 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.233638    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 01 00:33:53.233714 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.233675    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 01 00:33:53.235945 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.235922    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 01 00:33:53.263811 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.263790    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 01 00:33:53.264615 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.264602    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Aug 01 00:33:53.264689 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.264633    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Aug 01 00:33:53.264689 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.264648    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Aug 01 00:33:53.264689 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.264675    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-142.ec2.internal"
Aug 01 00:33:53.270984 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.270970    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-142.ec2.internal"
Aug 01 00:33:53.271041 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.270993    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-142.ec2.internal\": node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:53.285703 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:53.285627    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:53.333932 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.333909    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"]
Aug 01 00:33:53.334023 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.333964    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 01 00:33:53.334802 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.334786    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Aug 01 00:33:53.334875 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.334811    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Aug 01 00:33:53.334875 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.334820    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Aug 01 00:33:53.336074 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336062    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 01 00:33:53.336176 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336159    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:53.336226 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336191    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 01 00:33:53.336780 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336763    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Aug 01 00:33:53.336889 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336767    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Aug 01 00:33:53.336889 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336822    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Aug 01 00:33:53.336889 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336834    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Aug 01 00:33:53.336889 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336793    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Aug 01 00:33:53.336889 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.336879    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Aug 01 00:33:53.338181 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.338166    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:53.338224 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.338194    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 01 00:33:53.338849 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.338833    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientMemory"
Aug 01 00:33:53.338951 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.338856    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasNoDiskPressure"
Aug 01 00:33:53.338951 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:53.338881    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeHasSufficientPID"
Aug 01 00:33:54.394871 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:54.394853    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-142.ec2.internal\" not found" node="ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.399197 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:54.399181    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-142.ec2.internal\" not found" node="ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.408344 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:54.408330    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:54.430837 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.430814    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f923c99785cedeeb9d5a1e0081b46a58-config\") pod \"kube-apiserver-proxy-ip-10-0-141-142.ec2.internal\" (UID: \"f923c99785cedeeb9d5a1e0081b46a58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.430912 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.430840    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.430912 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.430856    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.509391 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:54.509360    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:54.531736 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.531717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.531811 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.531748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.531811 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.531776    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f923c99785cedeeb9d5a1e0081b46a58-config\") pod \"kube-apiserver-proxy-ip-10-0-141-142.ec2.internal\" (UID: \"f923c99785cedeeb9d5a1e0081b46a58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.531811 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.531802    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.531944 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.531801    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a631120ad11f3f853e9f18ea275f194-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal\" (UID: \"6a631120ad11f3f853e9f18ea275f194\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.531944 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.531831    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f923c99785cedeeb9d5a1e0081b46a58-config\") pod \"kube-apiserver-proxy-ip-10-0-141-142.ec2.internal\" (UID: \"f923c99785cedeeb9d5a1e0081b46a58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.610006 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:54.609952    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:54.697624 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.697602    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.701816 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:54.701710    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:54.710329 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:54.710310    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:54.810870 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:54.810847    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:54.911331 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:54.911277    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.011839 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:55.011809    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.075294 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.075273    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 01 00:33:55.075674 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.075412    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 01 00:33:55.112543 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:55.112521    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.147692 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.147665    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-31 00:28:54 +0000 UTC" deadline="2028-03-25 14:43:13.940443292 +0000 UTC"
Aug 01 00:33:55.147692 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.147689    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14462h9m18.792756603s"
Aug 01 00:33:55.213018 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:55.212970    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.313663 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:55.313643    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.320154 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:55.320115    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf923c99785cedeeb9d5a1e0081b46a58.slice/crio-dc89a7c0fd216fad3d3048f028ff72ce1a9dbf8899c706dd346c8c4673931381 WatchSource:0}: Error finding container dc89a7c0fd216fad3d3048f028ff72ce1a9dbf8899c706dd346c8c4673931381: Status 404 returned error can't find the container with id dc89a7c0fd216fad3d3048f028ff72ce1a9dbf8899c706dd346c8c4673931381
Aug 01 00:33:55.324140 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.324121    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:33:55.414457 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:55.414433    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.416716 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:55.416696    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a631120ad11f3f853e9f18ea275f194.slice/crio-b0737024ae3d57fc3ca54a94850c95bd97fa8c9df28abba934b085b4ee2c71d5 WatchSource:0}: Error finding container b0737024ae3d57fc3ca54a94850c95bd97fa8c9df28abba934b085b4ee2c71d5: Status 404 returned error can't find the container with id b0737024ae3d57fc3ca54a94850c95bd97fa8c9df28abba934b085b4ee2c71d5
Aug 01 00:33:55.501811 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.501760    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 01 00:33:55.514786 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:55.514764    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.615416 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:55.615395    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.687333 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.687306    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 01 00:33:55.716372 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:55.716349    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-142.ec2.internal\" not found"
Aug 01 00:33:55.718761 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.718739    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 01 00:33:55.728892 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.728866    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:55.738355 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.738341    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 01 00:33:55.739401 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.739387    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal"
Aug 01 00:33:55.747941 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:55.747912    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 01 00:33:56.112976 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.112946    2570 apiserver.go:52] "Watching apiserver"
Aug 01 00:33:56.120615 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.120586    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 01 00:33:56.121228 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.121194    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-wk8l6","kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px","openshift-dns/node-resolver-k8jhp","openshift-image-registry/node-ca-5fvlj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal","openshift-network-diagnostics/network-check-target-xxpkk","openshift-cluster-node-tuning-operator/tuned-wm8lb","openshift-multus/multus-59ckb","openshift-multus/multus-additional-cni-plugins-tq6z2","openshift-multus/network-metrics-daemon-99f8t","openshift-network-operator/iptables-alerter-5nfq8","openshift-ovn-kubernetes/ovnkube-node-qd5f8","kube-system/global-pull-secret-syncer-pgtlh"]
Aug 01 00:33:56.122911 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.122888    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:33:56.123027 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.122967    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:33:56.124006 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.123988    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.125461 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.125341    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.126585 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.126566    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 01 00:33:56.126727 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.126705    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 01 00:33:56.126857 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.126825    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.127885    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.128024    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.128039    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fw626\""
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.128027    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.128179    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.128193    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.128357    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.128397    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 01 00:33:56.128762 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.128546    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.130771 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.129916    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2x8zp\""
Aug 01 00:33:56.131361 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.131082    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.131361 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.131156    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:33:56.133283 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.131658    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.133283 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.132718    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 01 00:33:56.133283 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.132726    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bn95k\""
Aug 01 00:33:56.133283 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.132910    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 01 00:33:56.134969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.134114    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.134969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.134575    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 01 00:33:56.134969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.134585    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 01 00:33:56.135182 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.135023    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 01 00:33:56.135182 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.135038    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-42vns\""
Aug 01 00:33:56.136320 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.136155    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:33:56.136320 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.136274    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 01 00:33:56.136692 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.136672    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9dtsv\""
Aug 01 00:33:56.138094 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.137001    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 01 00:33:56.138094 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.137202    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 01 00:33:56.138094 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.137587    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 01 00:33:56.138094 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.137796    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.138560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.138543    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kvdvw\""
Aug 01 00:33:56.139150 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.138820    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 01 00:33:56.139274 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.138972    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.139401 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.139386    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 01 00:33:56.140041 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.140021    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 01 00:33:56.140129 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.140052    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:56.140187 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.140138    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:33:56.140187 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.140156    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6lfmv\""
Aug 01 00:33:56.140492 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.140473    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 01 00:33:56.141027 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141004    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 01 00:33:56.141371 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141352    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.141866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141680    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 01 00:33:56.141866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141713    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/00f870e8-987b-49cc-ab56-9af6d54510ed-ovn-node-metrics-cert\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.141866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trth8\" (UniqueName: \"kubernetes.io/projected/85d6afc8-b208-4fcf-b770-47f6560962a3-kube-api-access-trth8\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.141866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141769    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-dbus\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.141866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141790    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-slash\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.141866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141810    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-node-log\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.141866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141833    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.141866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141859    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141870    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-device-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141910    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7gp5x\""
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.141909    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-run-netns\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142032    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-etc-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142060    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142086    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-ovnkube-config\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142111    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142138    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad989299-a180-4b5d-ae39-9a5ffc294368-hosts-file\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142163    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85d6afc8-b208-4fcf-b770-47f6560962a3-serviceca\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-systemd\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142235    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-var-lib-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-ovn\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142293    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clsh7\" (UniqueName: \"kubernetes.io/projected/00f870e8-987b-49cc-ab56-9af6d54510ed-kube-api-access-clsh7\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142315    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-sys-fs\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142339    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvvx\" (UniqueName: \"kubernetes.io/projected/ad989299-a180-4b5d-ae39-9a5ffc294368-kube-api-access-jdvvx\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142367    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-kubelet-config\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142389    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-cni-netd\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142411    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhkt\" (UniqueName: \"kubernetes.io/projected/516de841-11e4-46d1-86c1-562eae8d2d65-kube-api-access-qrhkt\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142437    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142461    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-kubelet\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142482    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-ovnkube-script-lib\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142506    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-registration-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142531    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85d6afc8-b208-4fcf-b770-47f6560962a3-host\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142557    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142593    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142628    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-log-socket\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142656    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-env-overrides\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142686    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-socket-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad989299-a180-4b5d-ae39-9a5ffc294368-tmp-dir\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.142969 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142738    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-systemd-units\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.143708 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142759    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-etc-selinux\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.143708 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.142789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-cni-bin\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.143708 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.143687    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 01 00:33:56.143904 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.143886    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 01 00:33:56.143989 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.143926    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xjlc6\""
Aug 01 00:33:56.144189 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.144172    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 01 00:33:56.163577 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.163554    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xgqfc"
Aug 01 00:33:56.172747 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.172733    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xgqfc"
Aug 01 00:33:56.229284 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.229261    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 01 00:33:56.243443 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243404    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-k8s-cni-cncf-io\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.243560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243467    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-daemon-config\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.243560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243505    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-multus-certs\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.243560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243531    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/840dd47c-0006-4192-aa7e-db4a94a4b5ea-tmp\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.243560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243554    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2f9p\" (UniqueName: \"kubernetes.io/projected/840dd47c-0006-4192-aa7e-db4a94a4b5ea-kube-api-access-s2f9p\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.243749 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243581    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:33:56.243749 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243610    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-log-socket\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.243749 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-env-overrides\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.243749 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243718    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-log-socket\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.243947 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-socket-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.243947 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243860    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad989299-a180-4b5d-ae39-9a5ffc294368-tmp-dir\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.243947 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243888    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cni-binary-copy\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.243947 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243914    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysctl-conf\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.244132 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243949    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-host\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.244132 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-systemd-units\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244132 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.243995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-cnibin\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.244132 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244007    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-socket-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.244132 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244071    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-systemd-units\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244132 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244102    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-cni-multus\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.244132 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244127    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-modprobe-d\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244154    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-cni-bin\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244126    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-env-overrides\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244190    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad989299-a180-4b5d-ae39-9a5ffc294368-tmp-dir\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244235    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-cni-bin\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244177    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysctl-d\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244283    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-dbus\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244308    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-slash\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244349    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-slash\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244375    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-node-log\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244399    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-device-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244429    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6rq\" (UniqueName: \"kubernetes.io/projected/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-kube-api-access-bc6rq\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244454    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-run-netns\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244476    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-node-log\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.244484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244489    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-device-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244479    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-etc-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244515    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-dbus\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244521    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-run-netns\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244552    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-etc-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244584    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244607    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244630    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad989299-a180-4b5d-ae39-9a5ffc294368-hosts-file\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244670    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244682    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad989299-a180-4b5d-ae39-9a5ffc294368-hosts-file\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244701    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-sys-fs\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244728    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-kubernetes\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-systemd\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244757    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244776    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-sys-fs\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244780    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-cni-netd\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244816    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-cni-netd\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244821    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhkt\" (UniqueName: \"kubernetes.io/projected/516de841-11e4-46d1-86c1-562eae8d2d65-kube-api-access-qrhkt\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244853    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cnibin\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244877    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9rd\" (UniqueName: \"kubernetes.io/projected/31926079-d63f-43de-b52c-ced2e283ba10-kube-api-access-fc9rd\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244903    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-cni-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244942    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-hostroot\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysconfig\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.244994    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-ovnkube-script-lib\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-registration-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245065    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c68d149-59f9-4989-b7da-ae32080c74ba-konnectivity-ca\") pod \"konnectivity-agent-wk8l6\" (UID: \"9c68d149-59f9-4989-b7da-ae32080c74ba\") " pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245089    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-os-release\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245110    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-netns\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245132    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-var-lib-kubelet\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245159    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-system-cni-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245182    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85d6afc8-b208-4fcf-b770-47f6560962a3-host\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-system-cni-dir\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.245960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-lib-modules\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-etc-selinux\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245304    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245331    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5pj6\" (UniqueName: \"kubernetes.io/projected/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-kube-api-access-p5pj6\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245355    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c68d149-59f9-4989-b7da-ae32080c74ba-agent-certs\") pod \"konnectivity-agent-wk8l6\" (UID: \"9c68d149-59f9-4989-b7da-ae32080c74ba\") " pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245380    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngl2\" (UniqueName: \"kubernetes.io/projected/c3df4179-5885-44ad-a1c7-b7b585bb483f-kube-api-access-kngl2\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245399    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85d6afc8-b208-4fcf-b770-47f6560962a3-host\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245404    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-conf-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245442    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245445    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-run\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245468    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-sys\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245504    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/00f870e8-987b-49cc-ab56-9af6d54510ed-ovn-node-metrics-cert\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245528    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-os-release\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245534    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-ovnkube-script-lib\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245557    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-etc-selinux\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245574    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-cni-bin\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.246750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245330    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/516de841-11e4-46d1-86c1-562eae8d2d65-registration-dir\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245626    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trth8\" (UniqueName: \"kubernetes.io/projected/85d6afc8-b208-4fcf-b770-47f6560962a3-kube-api-access-trth8\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245665    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245744    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3df4179-5885-44ad-a1c7-b7b585bb483f-iptables-alerter-script\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245783    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-ovnkube-config\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245855    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-cni-binary-copy\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245880    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-kubelet\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245895    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245903    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85d6afc8-b208-4fcf-b770-47f6560962a3-serviceca\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.245957    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-systemd\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-var-lib-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246041    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-ovn\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246066    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clsh7\" (UniqueName: \"kubernetes.io/projected/00f870e8-987b-49cc-ab56-9af6d54510ed-kube-api-access-clsh7\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246095    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvvx\" (UniqueName: \"kubernetes.io/projected/ad989299-a180-4b5d-ae39-9a5ffc294368-kube-api-access-jdvvx\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246120    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3df4179-5885-44ad-a1c7-b7b585bb483f-host-slash\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.247411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246144    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-tuned\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246170    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-kubelet-config\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-socket-dir-parent\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246217    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-etc-kubernetes\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246266    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-kubelet\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246289    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/00f870e8-987b-49cc-ab56-9af6d54510ed-ovnkube-config\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246350    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-var-lib-openvswitch\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246360    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-ovn\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246358    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-run-systemd\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246393    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85d6afc8-b208-4fcf-b770-47f6560962a3-serviceca\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246397    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/00f870e8-987b-49cc-ab56-9af6d54510ed-host-kubelet\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.246426    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-kubelet-config\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.246448    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 01 00:33:56.248194 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.246589    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret podName:f5c55ca5-8428-4fbf-acbf-5c3d996c68d5 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:56.74656872 +0000 UTC m=+2.980500984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret") pod "global-pull-secret-syncer-pgtlh" (UID: "f5c55ca5-8428-4fbf-acbf-5c3d996c68d5") : object "kube-system"/"original-pull-secret" not registered
Aug 01 00:33:56.249546 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.249521    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 01 00:33:56.249546 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.249540    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 01 00:33:56.249679 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.249551    2570 projected.go:196] Error preparing data for projected volume kube-api-access-s2lz9 for pod openshift-network-diagnostics/network-check-target-xxpkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:33:56.249679 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.249628    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9 podName:1c57de4c-d851-4099-858a-550058fabda9 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:56.74961209 +0000 UTC m=+2.983544357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s2lz9" (UniqueName: "kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9") pod "network-check-target-xxpkk" (UID: "1c57de4c-d851-4099-858a-550058fabda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:33:56.251165 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.251140    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/00f870e8-987b-49cc-ab56-9af6d54510ed-ovn-node-metrics-cert\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.256846 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.256718    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trth8\" (UniqueName: \"kubernetes.io/projected/85d6afc8-b208-4fcf-b770-47f6560962a3-kube-api-access-trth8\") pod \"node-ca-5fvlj\" (UID: \"85d6afc8-b208-4fcf-b770-47f6560962a3\") " pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.256846 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.256758    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhkt\" (UniqueName: \"kubernetes.io/projected/516de841-11e4-46d1-86c1-562eae8d2d65-kube-api-access-qrhkt\") pod \"aws-ebs-csi-driver-node-sm7px\" (UID: \"516de841-11e4-46d1-86c1-562eae8d2d65\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.257058 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.257022    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clsh7\" (UniqueName: \"kubernetes.io/projected/00f870e8-987b-49cc-ab56-9af6d54510ed-kube-api-access-clsh7\") pod \"ovnkube-node-qd5f8\" (UID: \"00f870e8-987b-49cc-ab56-9af6d54510ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.257348 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.257306    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvvx\" (UniqueName: \"kubernetes.io/projected/ad989299-a180-4b5d-ae39-9a5ffc294368-kube-api-access-jdvvx\") pod \"node-resolver-k8jhp\" (UID: \"ad989299-a180-4b5d-ae39-9a5ffc294368\") " pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.260861 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.260818    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal" event={"ID":"6a631120ad11f3f853e9f18ea275f194","Type":"ContainerStarted","Data":"b0737024ae3d57fc3ca54a94850c95bd97fa8c9df28abba934b085b4ee2c71d5"}
Aug 01 00:33:56.262112 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.262087    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal" event={"ID":"f923c99785cedeeb9d5a1e0081b46a58","Type":"ContainerStarted","Data":"dc89a7c0fd216fad3d3048f028ff72ce1a9dbf8899c706dd346c8c4673931381"}
Aug 01 00:33:56.315540 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.315514    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 01 00:33:56.346514 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346492    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-kubernetes\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.346628 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346525    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-systemd\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.346628 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346550    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cnibin\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.346628 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9rd\" (UniqueName: \"kubernetes.io/projected/31926079-d63f-43de-b52c-ced2e283ba10-kube-api-access-fc9rd\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:56.346628 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346594    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-cni-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346619    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-systemd\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-hostroot\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346619    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-kubernetes\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346654    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysconfig\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346677    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-cni-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346685    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-hostroot\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346694    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysconfig\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346645    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cnibin\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346682    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346747    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c68d149-59f9-4989-b7da-ae32080c74ba-konnectivity-ca\") pod \"konnectivity-agent-wk8l6\" (UID: \"9c68d149-59f9-4989-b7da-ae32080c74ba\") " pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346773    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-os-release\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346799    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-netns\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.346836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346827    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346852    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-var-lib-kubelet\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346848    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-netns\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346874    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-os-release\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346897    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-system-cni-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-system-cni-dir\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346969    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-lib-modules\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346972    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-var-lib-kubelet\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346989    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-system-cni-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.346998    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347022    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-system-cni-dir\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347025    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5pj6\" (UniqueName: \"kubernetes.io/projected/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-kube-api-access-p5pj6\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347060    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c68d149-59f9-4989-b7da-ae32080c74ba-agent-certs\") pod \"konnectivity-agent-wk8l6\" (UID: \"9c68d149-59f9-4989-b7da-ae32080c74ba\") " pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347086    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngl2\" (UniqueName: \"kubernetes.io/projected/c3df4179-5885-44ad-a1c7-b7b585bb483f-kube-api-access-kngl2\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347108    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-conf-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347133    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-run\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347154    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-sys\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.347509 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347178    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-os-release\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-lib-modules\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347225    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-cni-bin\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347262    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-cni-bin\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347271    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347311    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-run\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347315    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3df4179-5885-44ad-a1c7-b7b585bb483f-iptables-alerter-script\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-cni-binary-copy\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347355    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-sys\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347367    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-kubelet\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3df4179-5885-44ad-a1c7-b7b585bb483f-host-slash\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347410    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-os-release\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c68d149-59f9-4989-b7da-ae32080c74ba-konnectivity-ca\") pod \"konnectivity-agent-wk8l6\" (UID: \"9c68d149-59f9-4989-b7da-ae32080c74ba\") " pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347428    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-conf-dir\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347474    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-kubelet\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.347490    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:33:56.348308 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.347540    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:56.847523051 +0000 UTC m=+3.081455310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347629    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347684    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3df4179-5885-44ad-a1c7-b7b585bb483f-host-slash\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347838    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-tuned\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347870    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-socket-dir-parent\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347894    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-etc-kubernetes\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347947    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-k8s-cni-cncf-io\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347972    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-daemon-config\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-multus-certs\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348019    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/840dd47c-0006-4192-aa7e-db4a94a4b5ea-tmp\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348033    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-cni-binary-copy\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2f9p\" (UniqueName: \"kubernetes.io/projected/840dd47c-0006-4192-aa7e-db4a94a4b5ea-kube-api-access-s2f9p\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348088    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-k8s-cni-cncf-io\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348088    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cni-binary-copy\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348118    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysctl-conf\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348140    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-host\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348157    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-cnibin\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348171    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-cni-multus\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-modprobe-d\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348205    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysctl-d\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348237    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6rq\" (UniqueName: \"kubernetes.io/projected/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-kube-api-access-bc6rq\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348475    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-host\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348494    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-cnibin\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348534    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-var-lib-cni-multus\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348539    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-socket-dir-parent\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.347907    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348577    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-etc-kubernetes\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348608    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-modprobe-d\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348613    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-host-run-multus-certs\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348681    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysctl-d\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348694    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-sysctl-conf\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.348750    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3df4179-5885-44ad-a1c7-b7b585bb483f-iptables-alerter-script\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.349791 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.349754    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-multus-daemon-config\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.350453 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.349851    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-cni-binary-copy\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.350453 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.350162    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c68d149-59f9-4989-b7da-ae32080c74ba-agent-certs\") pod \"konnectivity-agent-wk8l6\" (UID: \"9c68d149-59f9-4989-b7da-ae32080c74ba\") " pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:33:56.351771 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.351749    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/840dd47c-0006-4192-aa7e-db4a94a4b5ea-etc-tuned\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.352798 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.352310    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/840dd47c-0006-4192-aa7e-db4a94a4b5ea-tmp\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.356756 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.356731    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngl2\" (UniqueName: \"kubernetes.io/projected/c3df4179-5885-44ad-a1c7-b7b585bb483f-kube-api-access-kngl2\") pod \"iptables-alerter-5nfq8\" (UID: \"c3df4179-5885-44ad-a1c7-b7b585bb483f\") " pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.356871 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.356754    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5pj6\" (UniqueName: \"kubernetes.io/projected/c2ff6e14-3a7a-4d02-a3c1-5d4e34712996-kube-api-access-p5pj6\") pod \"multus-additional-cni-plugins-tq6z2\" (UID: \"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996\") " pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.357157 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.357133    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9rd\" (UniqueName: \"kubernetes.io/projected/31926079-d63f-43de-b52c-ced2e283ba10-kube-api-access-fc9rd\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:56.357377 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.357358    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6rq\" (UniqueName: \"kubernetes.io/projected/4efd71b3-8bef-4b32-ae27-5d1e83c2d393-kube-api-access-bc6rq\") pod \"multus-59ckb\" (UID: \"4efd71b3-8bef-4b32-ae27-5d1e83c2d393\") " pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.358277 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.358253    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2f9p\" (UniqueName: \"kubernetes.io/projected/840dd47c-0006-4192-aa7e-db4a94a4b5ea-kube-api-access-s2f9p\") pod \"tuned-wm8lb\" (UID: \"840dd47c-0006-4192-aa7e-db4a94a4b5ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.440350 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.440287    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:33:56.448913 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.448888    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px"
Aug 01 00:33:56.457514 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.457498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k8jhp"
Aug 01 00:33:56.466879 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.466861    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5fvlj"
Aug 01 00:33:56.472519 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.472504    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-59ckb"
Aug 01 00:33:56.480904 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.480884    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:33:56.488513 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.488492    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wm8lb"
Aug 01 00:33:56.497159 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.497142    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tq6z2"
Aug 01 00:33:56.506619 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.506604    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5nfq8"
Aug 01 00:33:56.750960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.750867    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:56.750960 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.750901    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:33:56.751178 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.751030    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 01 00:33:56.751178 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.751051    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 01 00:33:56.751178 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.751065    2570 projected.go:196] Error preparing data for projected volume kube-api-access-s2lz9 for pod openshift-network-diagnostics/network-check-target-xxpkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:33:56.751178 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.751034    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 01 00:33:56.751178 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.751112    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9 podName:1c57de4c-d851-4099-858a-550058fabda9 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:57.75109471 +0000 UTC m=+3.985026954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2lz9" (UniqueName: "kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9") pod "network-check-target-xxpkk" (UID: "1c57de4c-d851-4099-858a-550058fabda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:33:56.751364 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.751183    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret podName:f5c55ca5-8428-4fbf-acbf-5c3d996c68d5 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:57.751162873 +0000 UTC m=+3.985095124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret") pod "global-pull-secret-syncer-pgtlh" (UID: "f5c55ca5-8428-4fbf-acbf-5c3d996c68d5") : object "kube-system"/"original-pull-secret" not registered
Aug 01 00:33:56.851313 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:56.851275    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:56.851471 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.851426    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:33:56.851519 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:56.851489    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:57.851471347 +0000 UTC m=+4.085403591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:33:57.163312 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.163288    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad989299_a180_4b5d_ae39_9a5ffc294368.slice/crio-b4bbc3b85a0b9cddf06b8cc1d783842bf4c0e962d7c4bbda7a865085e43a8f11 WatchSource:0}: Error finding container b4bbc3b85a0b9cddf06b8cc1d783842bf4c0e962d7c4bbda7a865085e43a8f11: Status 404 returned error can't find the container with id b4bbc3b85a0b9cddf06b8cc1d783842bf4c0e962d7c4bbda7a865085e43a8f11
Aug 01 00:33:57.164581 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.164563    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516de841_11e4_46d1_86c1_562eae8d2d65.slice/crio-1e25052798e0610ca399b36918b6a3871770d66fbd26c20bbc52da092e7dc944 WatchSource:0}: Error finding container 1e25052798e0610ca399b36918b6a3871770d66fbd26c20bbc52da092e7dc944: Status 404 returned error can't find the container with id 1e25052798e0610ca399b36918b6a3871770d66fbd26c20bbc52da092e7dc944
Aug 01 00:33:57.166193 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.166175    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d6afc8_b208_4fcf_b770_47f6560962a3.slice/crio-f269935dddaafabaf6f3cec1540462159aad57f11f467260818805f4d2bb4ede WatchSource:0}: Error finding container f269935dddaafabaf6f3cec1540462159aad57f11f467260818805f4d2bb4ede: Status 404 returned error can't find the container with id f269935dddaafabaf6f3cec1540462159aad57f11f467260818805f4d2bb4ede
Aug 01 00:33:57.169959 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.169915    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840dd47c_0006_4192_aa7e_db4a94a4b5ea.slice/crio-cf72a1d1a044c7c1517e7d9f93925fa3db3237c075f7ee40e614df556d4da102 WatchSource:0}: Error finding container cf72a1d1a044c7c1517e7d9f93925fa3db3237c075f7ee40e614df556d4da102: Status 404 returned error can't find the container with id cf72a1d1a044c7c1517e7d9f93925fa3db3237c075f7ee40e614df556d4da102
Aug 01 00:33:57.171452 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.171434    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efd71b3_8bef_4b32_ae27_5d1e83c2d393.slice/crio-4d5125d8accf84b23a5dce4d59b2b11ce216750dd013d51163c9c01b396d5196 WatchSource:0}: Error finding container 4d5125d8accf84b23a5dce4d59b2b11ce216750dd013d51163c9c01b396d5196: Status 404 returned error can't find the container with id 4d5125d8accf84b23a5dce4d59b2b11ce216750dd013d51163c9c01b396d5196
Aug 01 00:33:57.173033 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.173010    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ff6e14_3a7a_4d02_a3c1_5d4e34712996.slice/crio-70154b816c06dd93efc3ed1f9296f34ed592673a917f5639f2f360b268f03ef0 WatchSource:0}: Error finding container 70154b816c06dd93efc3ed1f9296f34ed592673a917f5639f2f360b268f03ef0: Status 404 returned error can't find the container with id 70154b816c06dd93efc3ed1f9296f34ed592673a917f5639f2f360b268f03ef0
Aug 01 00:33:57.173890 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.173866    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f870e8_987b_49cc_ab56_9af6d54510ed.slice/crio-9f946fc56b40c39a09ec9aa48d660c16b9b5297c4986e3ee1f307a541d518c5b WatchSource:0}: Error finding container 9f946fc56b40c39a09ec9aa48d660c16b9b5297c4986e3ee1f307a541d518c5b: Status 404 returned error can't find the container with id 9f946fc56b40c39a09ec9aa48d660c16b9b5297c4986e3ee1f307a541d518c5b
Aug 01 00:33:57.175113 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.175096    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c68d149_59f9_4989_b7da_ae32080c74ba.slice/crio-9049d5cfae389c7f909cf9fd8b0c1085a2d86d3a7dc341803f9471e83b0d5224 WatchSource:0}: Error finding container 9049d5cfae389c7f909cf9fd8b0c1085a2d86d3a7dc341803f9471e83b0d5224: Status 404 returned error can't find the container with id 9049d5cfae389c7f909cf9fd8b0c1085a2d86d3a7dc341803f9471e83b0d5224
Aug 01 00:33:57.175752 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.175692    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-31 00:28:56 +0000 UTC" deadline="2028-01-06 06:30:26.574285584 +0000 UTC"
Aug 01 00:33:57.175752 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.175751    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12557h56m29.398538959s"
Aug 01 00:33:57.176728 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:33:57.176237    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3df4179_5885_44ad_a1c7_b7b585bb483f.slice/crio-8812a6046f94f92d1a7e0ff8a098bb8f088614b50014e37b783b3346e2692ee3 WatchSource:0}: Error finding container 8812a6046f94f92d1a7e0ff8a098bb8f088614b50014e37b783b3346e2692ee3: Status 404 returned error can't find the container with id 8812a6046f94f92d1a7e0ff8a098bb8f088614b50014e37b783b3346e2692ee3
Aug 01 00:33:57.269106 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.269073    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wm8lb" event={"ID":"840dd47c-0006-4192-aa7e-db4a94a4b5ea","Type":"ContainerStarted","Data":"cf72a1d1a044c7c1517e7d9f93925fa3db3237c075f7ee40e614df556d4da102"}
Aug 01 00:33:57.271280 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.271254    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5fvlj" event={"ID":"85d6afc8-b208-4fcf-b770-47f6560962a3","Type":"ContainerStarted","Data":"f269935dddaafabaf6f3cec1540462159aad57f11f467260818805f4d2bb4ede"}
Aug 01 00:33:57.273195 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.273174    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k8jhp" event={"ID":"ad989299-a180-4b5d-ae39-9a5ffc294368","Type":"ContainerStarted","Data":"b4bbc3b85a0b9cddf06b8cc1d783842bf4c0e962d7c4bbda7a865085e43a8f11"}
Aug 01 00:33:57.274131 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.274108    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" event={"ID":"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996","Type":"ContainerStarted","Data":"70154b816c06dd93efc3ed1f9296f34ed592673a917f5639f2f360b268f03ef0"}
Aug 01 00:33:57.275047 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.275025    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px" event={"ID":"516de841-11e4-46d1-86c1-562eae8d2d65","Type":"ContainerStarted","Data":"1e25052798e0610ca399b36918b6a3871770d66fbd26c20bbc52da092e7dc944"}
Aug 01 00:33:57.276222 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.276194    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5nfq8" event={"ID":"c3df4179-5885-44ad-a1c7-b7b585bb483f","Type":"ContainerStarted","Data":"8812a6046f94f92d1a7e0ff8a098bb8f088614b50014e37b783b3346e2692ee3"}
Aug 01 00:33:57.278491 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.278473    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wk8l6" event={"ID":"9c68d149-59f9-4989-b7da-ae32080c74ba","Type":"ContainerStarted","Data":"9049d5cfae389c7f909cf9fd8b0c1085a2d86d3a7dc341803f9471e83b0d5224"}
Aug 01 00:33:57.284146 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.284125    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"9f946fc56b40c39a09ec9aa48d660c16b9b5297c4986e3ee1f307a541d518c5b"}
Aug 01 00:33:57.285411 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.285391    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-59ckb" event={"ID":"4efd71b3-8bef-4b32-ae27-5d1e83c2d393","Type":"ContainerStarted","Data":"4d5125d8accf84b23a5dce4d59b2b11ce216750dd013d51163c9c01b396d5196"}
Aug 01 00:33:57.756972 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.756384    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:57.756972 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.756655    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:33:57.756972 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:57.756730    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 01 00:33:57.756972 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:57.756785    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 01 00:33:57.756972 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:57.756801    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 01 00:33:57.756972 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:57.756811    2570 projected.go:196] Error preparing data for projected volume kube-api-access-s2lz9 for pod openshift-network-diagnostics/network-check-target-xxpkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:33:57.756972 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:57.756787    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret podName:f5c55ca5-8428-4fbf-acbf-5c3d996c68d5 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:59.756769627 +0000 UTC m=+5.990701875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret") pod "global-pull-secret-syncer-pgtlh" (UID: "f5c55ca5-8428-4fbf-acbf-5c3d996c68d5") : object "kube-system"/"original-pull-secret" not registered
Aug 01 00:33:57.756972 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:57.756851    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9 podName:1c57de4c-d851-4099-858a-550058fabda9 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:59.75684105 +0000 UTC m=+5.990773296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2lz9" (UniqueName: "kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9") pod "network-check-target-xxpkk" (UID: "1c57de4c-d851-4099-858a-550058fabda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:33:57.857892 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:57.857380    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:57.857892 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:57.857543    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:33:57.857892 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:57.857596    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:59.857578082 +0000 UTC m=+6.091510330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:33:58.176153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.176069    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-31 00:28:56 +0000 UTC" deadline="2028-04-15 04:19:53.454450778 +0000 UTC"
Aug 01 00:33:58.176153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.176107    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14955h45m55.278348404s"
Aug 01 00:33:58.266093 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.266051    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:58.266249 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:58.266225    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:33:58.266733 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.266712    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:58.266917 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:58.266894    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:33:58.267380 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.267361    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:33:58.267507 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:58.267487    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:33:58.299125 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.299094    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal" event={"ID":"6a631120ad11f3f853e9f18ea275f194","Type":"ContainerDied","Data":"72619612276ea89856c0bc0c52bbe88f156b90448b50deea8be86b10439e262c"}
Aug 01 00:33:58.299430 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.299405    2570 generic.go:356] "Generic (PLEG): container finished" podID="6a631120ad11f3f853e9f18ea275f194" containerID="72619612276ea89856c0bc0c52bbe88f156b90448b50deea8be86b10439e262c" exitCode=0
Aug 01 00:33:58.326174 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.326112    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal" event={"ID":"f923c99785cedeeb9d5a1e0081b46a58","Type":"ContainerStarted","Data":"ac37101654baf41f425ec719dc67ca3a86338afd792b4571c711eeefbea7e872"}
Aug 01 00:33:58.340065 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:58.339800    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-142.ec2.internal" podStartSLOduration=3.339783835 podStartE2EDuration="3.339783835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:58.339576691 +0000 UTC m=+4.573508957" watchObservedRunningTime="2026-08-01 00:33:58.339783835 +0000 UTC m=+4.573716101"
Aug 01 00:33:59.338282 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:59.338245    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal" event={"ID":"6a631120ad11f3f853e9f18ea275f194","Type":"ContainerStarted","Data":"7c766d92c9c9a69680b4c5e7646e71cb099580b1dbf9ad9068b6f077a1723303"}
Aug 01 00:33:59.780361 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:59.780282    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:33:59.780361 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:59.780335    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:33:59.780567 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:59.780539    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 01 00:33:59.780567 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:59.780561    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 01 00:33:59.780675 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:59.780573    2570 projected.go:196] Error preparing data for projected volume kube-api-access-s2lz9 for pod openshift-network-diagnostics/network-check-target-xxpkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:33:59.780675 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:59.780629    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9 podName:1c57de4c-d851-4099-858a-550058fabda9 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:03.780611533 +0000 UTC m=+10.014543782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2lz9" (UniqueName: "kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9") pod "network-check-target-xxpkk" (UID: "1c57de4c-d851-4099-858a-550058fabda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:33:59.781071 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:59.781047    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 01 00:33:59.781166 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:59.781115    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret podName:f5c55ca5-8428-4fbf-acbf-5c3d996c68d5 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:03.781098019 +0000 UTC m=+10.015030279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret") pod "global-pull-secret-syncer-pgtlh" (UID: "f5c55ca5-8428-4fbf-acbf-5c3d996c68d5") : object "kube-system"/"original-pull-secret" not registered
Aug 01 00:33:59.881090 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:33:59.881052    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:33:59.881247 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:59.881197    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:33:59.881317 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:33:59.881255    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:03.881236649 +0000 UTC m=+10.115168895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:34:00.256958 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:00.256411    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:00.256958 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:00.256420    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:00.256958 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:00.256544    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:00.256958 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:00.256611    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:00.256958 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:00.256725    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:00.256958 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:00.256817    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:02.257058 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:02.257023    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:02.257516 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:02.257154    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:02.257516 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:02.257170    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:02.257516 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:02.257274    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:02.257516 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:02.257025    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:02.257516 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:02.257376    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:03.816343 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:03.816301    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:03.816852 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:03.816357    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:03.816852 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:03.816504    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 01 00:34:03.816852 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:03.816524    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 01 00:34:03.816852 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:03.816537    2570 projected.go:196] Error preparing data for projected volume kube-api-access-s2lz9 for pod openshift-network-diagnostics/network-check-target-xxpkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:34:03.816852 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:03.816595    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9 podName:1c57de4c-d851-4099-858a-550058fabda9 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:11.816575872 +0000 UTC m=+18.050508138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2lz9" (UniqueName: "kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9") pod "network-check-target-xxpkk" (UID: "1c57de4c-d851-4099-858a-550058fabda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:34:03.817204 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:03.817021    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 01 00:34:03.817204 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:03.817080    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret podName:f5c55ca5-8428-4fbf-acbf-5c3d996c68d5 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:11.817062805 +0000 UTC m=+18.050995052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret") pod "global-pull-secret-syncer-pgtlh" (UID: "f5c55ca5-8428-4fbf-acbf-5c3d996c68d5") : object "kube-system"/"original-pull-secret" not registered
Aug 01 00:34:03.916801 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:03.916759    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:03.916986 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:03.916909    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:34:03.917036 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:03.917006    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:11.916984549 +0000 UTC m=+18.150916806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:34:04.256897 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:04.256814    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:04.257060 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:04.256904    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:04.257060 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:04.256956    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:04.257060 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:04.257027    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:04.257224 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:04.257066    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:04.257224 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:04.257135    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:06.256462 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:06.256142    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:06.256462 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:06.256245    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:06.256462 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:06.256314    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:06.256462 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:06.256372    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:06.256462 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:06.256404    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:06.256462 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:06.256442    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:08.256095 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:08.256061    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:08.256095 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:08.256097    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:08.256597 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:08.256064    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:08.256597 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:08.256182    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:08.256597 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:08.256256    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:08.256597 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:08.256344    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:10.255998 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:10.255965    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:10.256488 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:10.255971    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:10.256488 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:10.256095    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:10.256488 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:10.255971    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:10.256488 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:10.256233    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:10.256488 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:10.256342    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:11.880661 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:11.880632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:11.881161 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:11.880670    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:11.881161 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:11.880788    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 01 00:34:11.881161 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:11.880805    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 01 00:34:11.881161 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:11.880836    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 01 00:34:11.881161 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:11.880849    2570 projected.go:196] Error preparing data for projected volume kube-api-access-s2lz9 for pod openshift-network-diagnostics/network-check-target-xxpkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:34:11.881161 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:11.880871    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret podName:f5c55ca5-8428-4fbf-acbf-5c3d996c68d5 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:27.880851629 +0000 UTC m=+34.114783872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret") pod "global-pull-secret-syncer-pgtlh" (UID: "f5c55ca5-8428-4fbf-acbf-5c3d996c68d5") : object "kube-system"/"original-pull-secret" not registered
Aug 01 00:34:11.881161 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:11.880894    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9 podName:1c57de4c-d851-4099-858a-550058fabda9 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:27.880880428 +0000 UTC m=+34.114812672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2lz9" (UniqueName: "kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9") pod "network-check-target-xxpkk" (UID: "1c57de4c-d851-4099-858a-550058fabda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:34:11.981191 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:11.981159    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:11.981422 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:11.981274    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:34:11.981422 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:11.981335    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:27.981314616 +0000 UTC m=+34.215246861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:34:12.256398 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:12.256321    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:12.256398 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:12.256355    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:12.256610 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:12.256321    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:12.256610 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:12.256473    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:12.256610 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:12.256552    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:12.256715 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:12.256662    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:14.257083 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:14.257048    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:14.257083 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:14.257082    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:14.257615 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:14.257149    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:14.257615 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:14.257206    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:14.257615 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:14.257273    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:14.257615 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:14.257374    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:16.256520 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.256349    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:16.257113 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.256356    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:16.257113 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:16.256600    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:16.257113 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:16.256661    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:16.257113 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.256362    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:16.257113 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:16.256734    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:16.370645 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.370532    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wk8l6" event={"ID":"9c68d149-59f9-4989-b7da-ae32080c74ba","Type":"ContainerStarted","Data":"daa6fb65aab7660c3e19755c28162666e7a157451c4481e1707e6f4120824155"}
Aug 01 00:34:16.373721 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.373677    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"fa5c6a95dd60d33a2eba095f1bcd951306d63de83b66e05d535d9233147b1088"}
Aug 01 00:34:16.373824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.373755    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"0c143a37b850199a1bbaf657e690930910ebb9ee2ac03e5e66c5ae7d9c44c689"}
Aug 01 00:34:16.373888 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.373814    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"37e31ac53c90d50087555199b89bbd806ae288da1e7f907874a193ae66a267d5"}
Aug 01 00:34:16.373966 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.373889    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"c0fcdc242d7a58f38eed90a12d2dc988d70967c596c72d7a7019e5d121fb19ac"}
Aug 01 00:34:16.373966 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.373961    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"fa36f8fc8c3426ded9b66f9acf5af6641aebe0e8ae1a565ab6f20a39ab14c13f"}
Aug 01 00:34:16.376055 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.376033    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-59ckb" event={"ID":"4efd71b3-8bef-4b32-ae27-5d1e83c2d393","Type":"ContainerStarted","Data":"7f170db332cb326e3e285526fa058dcf8963e9c6758f68cebd20f0e588c55c5f"}
Aug 01 00:34:16.377888 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.377861    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wm8lb" event={"ID":"840dd47c-0006-4192-aa7e-db4a94a4b5ea","Type":"ContainerStarted","Data":"2f9720aec13301820d7e13e5218252f670eeb45a26d38b0cf0624845dc89e528"}
Aug 01 00:34:16.379323 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.379302    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5fvlj" event={"ID":"85d6afc8-b208-4fcf-b770-47f6560962a3","Type":"ContainerStarted","Data":"3cfcd389988cc5a84e55243680cd6fcf2482c4b818a47659d0a4424bdcf166e8"}
Aug 01 00:34:16.380755 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.380721    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k8jhp" event={"ID":"ad989299-a180-4b5d-ae39-9a5ffc294368","Type":"ContainerStarted","Data":"5a545cb3e54c98ab348a1ef0907e10f9f0698cb34f16a70a331dde770e83b569"}
Aug 01 00:34:16.382193 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.382173    2570 generic.go:356] "Generic (PLEG): container finished" podID="c2ff6e14-3a7a-4d02-a3c1-5d4e34712996" containerID="50a20fc6f444f578bfaada9abb3c24e324f19784696eb775bd9bdc37578f8aa2" exitCode=0
Aug 01 00:34:16.382279 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.382237    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" event={"ID":"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996","Type":"ContainerDied","Data":"50a20fc6f444f578bfaada9abb3c24e324f19784696eb775bd9bdc37578f8aa2"}
Aug 01 00:34:16.383755 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.383736    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px" event={"ID":"516de841-11e4-46d1-86c1-562eae8d2d65","Type":"ContainerStarted","Data":"ab99efd089b97b63d39fe22969a7a7f85b5a1548d15c14e14e0ef04c44bdd68e"}
Aug 01 00:34:16.385193 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.385159    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wk8l6" podStartSLOduration=8.077177353 podStartE2EDuration="22.385146831s" totalImagesPullingTime="14.307969478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:16.384875881 +0000 UTC m=+22.618808145" watchObservedRunningTime="2026-08-01 00:34:16.385146831 +0000 UTC m=+22.619079096"
Aug 01 00:34:16.385415 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.385391    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-142.ec2.internal" podStartSLOduration=21.38538449 podStartE2EDuration="21.38538449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:59.354475587 +0000 UTC m=+5.588407854" watchObservedRunningTime="2026-08-01 00:34:16.38538449 +0000 UTC m=+22.619316755"
Aug 01 00:34:16.400625 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.400590    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5fvlj" podStartSLOduration=4.023111133 podStartE2EDuration="22.400581358s" totalImagesPullingTime="18.377470225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:16.400280127 +0000 UTC m=+22.634212396" watchObservedRunningTime="2026-08-01 00:34:16.400581358 +0000 UTC m=+22.634513646"
Aug 01 00:34:16.417850 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.417783    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-59ckb" podStartSLOduration=3.9217924440000003 podStartE2EDuration="22.417771157s" totalImagesPullingTime="18.495978713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:16.417751734 +0000 UTC m=+22.651684014" watchObservedRunningTime="2026-08-01 00:34:16.417771157 +0000 UTC m=+22.651703416"
Aug 01 00:34:16.433592 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.433517    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wm8lb" podStartSLOduration=3.98213296 podStartE2EDuration="22.433499888s" totalImagesPullingTime="18.451366928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:16.433029051 +0000 UTC m=+22.666961316" watchObservedRunningTime="2026-08-01 00:34:16.433499888 +0000 UTC m=+22.667432156"
Aug 01 00:34:16.471537 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.471501    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k8jhp" podStartSLOduration=4.013150952 podStartE2EDuration="22.471487175s" totalImagesPullingTime="18.458336223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:16.471112839 +0000 UTC m=+22.705045115" watchObservedRunningTime="2026-08-01 00:34:16.471487175 +0000 UTC m=+22.705419439"
Aug 01 00:34:16.930087 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:16.930027    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 01 00:34:17.200858 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:17.200719    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-01T00:34:16.93004336Z","UUID":"e073e030-98d8-4351-ae76-536c22854af0","Handler":null,"Name":"","Endpoint":""}
Aug 01 00:34:17.202069 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:17.202050    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 01 00:34:17.202189 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:17.202091    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 01 00:34:17.387591 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:17.387553    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px" event={"ID":"516de841-11e4-46d1-86c1-562eae8d2d65","Type":"ContainerStarted","Data":"991b6f8ce04dda9568d4f6b56334fe3fd1bb02306abab505b6393a15995f7eae"}
Aug 01 00:34:17.389520 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:17.389205    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5nfq8" event={"ID":"c3df4179-5885-44ad-a1c7-b7b585bb483f","Type":"ContainerStarted","Data":"5adc710395edf7b0a1fba44c96c06684480f8952b6f0974cd4f450ecaa3c157f"}
Aug 01 00:34:17.392346 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:17.392317    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"89f9856e07c9009b29cb9c128e3c1a24fc0e9362560d7dd264a940db8c01969f"}
Aug 01 00:34:17.405101 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:17.404786    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5nfq8" podStartSLOduration=5.037161774 podStartE2EDuration="23.404769549s" totalImagesPullingTime="18.367607775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:17.404060326 +0000 UTC m=+23.637992593" watchObservedRunningTime="2026-08-01 00:34:17.404769549 +0000 UTC m=+23.638701817"
Aug 01 00:34:18.256968 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:18.256881    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:18.256968 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:18.256909    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:18.256968 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:18.256889    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:18.257214 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:18.257039    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:18.257214 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:18.257118    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:18.257214 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:18.257189    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:18.396502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:18.396471    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px" event={"ID":"516de841-11e4-46d1-86c1-562eae8d2d65","Type":"ContainerStarted","Data":"4a8c73457822e445d488873a4b34ee115fd76099513c4084482b169969ddc3af"}
Aug 01 00:34:18.420557 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:18.420519    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sm7px" podStartSLOduration=3.9915125209999998 podStartE2EDuration="24.420504311s" totalImagesPullingTime="20.42899179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:18.419328011 +0000 UTC m=+24.653260276" watchObservedRunningTime="2026-08-01 00:34:18.420504311 +0000 UTC m=+24.654436574"
Aug 01 00:34:18.686909 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:18.686883    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:34:18.687621 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:18.687588    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:34:19.402671 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:19.402470    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"fc88e0c260a196912a3d755b88eae18b8547cf8064b9af214e1820755c5ec18e"}
Aug 01 00:34:19.403136 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:19.402873    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:34:19.403209 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:19.403188    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wk8l6"
Aug 01 00:34:20.257011 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:20.256985    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:20.257209 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:20.257093    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:20.257209 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:20.256981    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:20.257209 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:20.257122    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:20.257370 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:20.257215    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:20.257370 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:20.257316    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:21.410749 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:21.410557    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" event={"ID":"00f870e8-987b-49cc-ab56-9af6d54510ed","Type":"ContainerStarted","Data":"5400696c2a9448d70251a8ab090344f049475884eebd685e9a93a4c9b70618ec"}
Aug 01 00:34:21.436995 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:21.436957    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8" podStartSLOduration=8.759762548 podStartE2EDuration="27.436924667s" totalImagesPullingTime="18.677162119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:21.435002197 +0000 UTC m=+27.668934461" watchObservedRunningTime="2026-08-01 00:34:21.436924667 +0000 UTC m=+27.670856931"
Aug 01 00:34:22.256107 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.256071    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:22.256286 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.256071    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:22.256286 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:22.256168    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:22.256286 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.256176    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:22.256286 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:22.256254    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:22.256516 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:22.256368    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:22.416636 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.416520    2570 generic.go:356] "Generic (PLEG): container finished" podID="c2ff6e14-3a7a-4d02-a3c1-5d4e34712996" containerID="1349610226f5778f3edcbccab3088794bb6c58ad2b6ba191e476eb0653c941d5" exitCode=0
Aug 01 00:34:22.418108 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.417413    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" event={"ID":"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996","Type":"ContainerDied","Data":"1349610226f5778f3edcbccab3088794bb6c58ad2b6ba191e476eb0653c941d5"}
Aug 01 00:34:22.418108 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.417842    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:34:22.418108 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.417957    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:34:22.418108 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.418003    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:34:22.444117 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.443890    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:34:22.453756 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:22.453640    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:34:23.466309 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:23.466279    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xxpkk"]
Aug 01 00:34:23.466699 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:23.466389    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:23.466699 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:23.466471    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:23.469107 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:23.469082    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pgtlh"]
Aug 01 00:34:23.469233 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:23.469187    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:23.469289 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:23.469270    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:23.469671 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:23.469648    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-99f8t"]
Aug 01 00:34:23.469767 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:23.469753    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:23.469858 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:23.469839    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:24.421697 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:24.421666    2570 generic.go:356] "Generic (PLEG): container finished" podID="c2ff6e14-3a7a-4d02-a3c1-5d4e34712996" containerID="1c94bd839e1ebfad001241c4455fdecaa074d24aa2c782de60f90b6ec4cb0dfd" exitCode=0
Aug 01 00:34:24.421842 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:24.421753    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" event={"ID":"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996","Type":"ContainerDied","Data":"1c94bd839e1ebfad001241c4455fdecaa074d24aa2c782de60f90b6ec4cb0dfd"}
Aug 01 00:34:25.256267 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:25.256233    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:25.256267 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:25.256261    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:25.256748 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:25.256277    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:25.256748 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:25.256375    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:25.256748 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:25.256512    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:25.256748 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:25.256602    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:26.430538 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:26.430502    2570 generic.go:356] "Generic (PLEG): container finished" podID="c2ff6e14-3a7a-4d02-a3c1-5d4e34712996" containerID="6979dfc6e4e086154679e6a12d3bec8771a8bc9685473618da35c01a408c1343" exitCode=0
Aug 01 00:34:26.431101 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:26.430548    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" event={"ID":"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996","Type":"ContainerDied","Data":"6979dfc6e4e086154679e6a12d3bec8771a8bc9685473618da35c01a408c1343"}
Aug 01 00:34:27.256548 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:27.256470    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:27.256705 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:27.256582    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:27.256705 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.256591    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pgtlh" podUID="f5c55ca5-8428-4fbf-acbf-5c3d996c68d5"
Aug 01 00:34:27.256705 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:27.256616    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:27.256848 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.256702    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxpkk" podUID="1c57de4c-d851-4099-858a-550058fabda9"
Aug 01 00:34:27.256884 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.256848    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:34:27.903433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:27.903394    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:27.903433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:27.903439    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:27.904098 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.903550    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 01 00:34:27.904098 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.903624    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret podName:f5c55ca5-8428-4fbf-acbf-5c3d996c68d5 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:59.903602321 +0000 UTC m=+66.137534566 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret") pod "global-pull-secret-syncer-pgtlh" (UID: "f5c55ca5-8428-4fbf-acbf-5c3d996c68d5") : object "kube-system"/"original-pull-secret" not registered
Aug 01 00:34:27.904098 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.903558    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 01 00:34:27.904098 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.903680    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 01 00:34:27.904098 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.903695    2570 projected.go:196] Error preparing data for projected volume kube-api-access-s2lz9 for pod openshift-network-diagnostics/network-check-target-xxpkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:34:27.904098 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:27.903729    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9 podName:1c57de4c-d851-4099-858a-550058fabda9 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:59.903719798 +0000 UTC m=+66.137652043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2lz9" (UniqueName: "kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9") pod "network-check-target-xxpkk" (UID: "1c57de4c-d851-4099-858a-550058fabda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 01 00:34:28.004357 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.004324    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:28.004521 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:28.004479    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:34:28.004583 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:28.004548    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:35:00.0045279 +0000 UTC m=+66.238460149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 01 00:34:28.609631 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.609600    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-142.ec2.internal" event="NodeReady"
Aug 01 00:34:28.609792 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.609737    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 01 00:34:28.654920 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.654856    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r7rdc"]
Aug 01 00:34:28.657149 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.657126    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5w4jn"]
Aug 01 00:34:28.657277 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.657261    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.658829 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.658810    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5w4jn"
Aug 01 00:34:28.659874 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.659854    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 01 00:34:28.659988 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.659860    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 01 00:34:28.659988 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.659923    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qd5dz\""
Aug 01 00:34:28.661537 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.661373    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 01 00:34:28.661537 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.661417    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 01 00:34:28.661537 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.661433    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-msh2l\""
Aug 01 00:34:28.661537 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.661468    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 01 00:34:28.666651 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.666631    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r7rdc"]
Aug 01 00:34:28.679221 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.679180    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5w4jn"]
Aug 01 00:34:28.808776 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.808745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d36e3319-9c8a-480e-aa41-4e7a98c0532a-tmp-dir\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.808967 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.808787    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57prl\" (UniqueName: \"kubernetes.io/projected/a8d42714-2762-4696-8f20-c1005f745797-kube-api-access-57prl\") pod \"ingress-canary-5w4jn\" (UID: \"a8d42714-2762-4696-8f20-c1005f745797\") " pod="openshift-ingress-canary/ingress-canary-5w4jn"
Aug 01 00:34:28.808967 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.808894    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d36e3319-9c8a-480e-aa41-4e7a98c0532a-config-volume\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.808967 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.808917    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8d42714-2762-4696-8f20-c1005f745797-cert\") pod \"ingress-canary-5w4jn\" (UID: \"a8d42714-2762-4696-8f20-c1005f745797\") " pod="openshift-ingress-canary/ingress-canary-5w4jn"
Aug 01 00:34:28.808967 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.808952    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.809138 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.808987    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdsjc\" (UniqueName: \"kubernetes.io/projected/d36e3319-9c8a-480e-aa41-4e7a98c0532a-kube-api-access-xdsjc\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.909507 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.909432    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d36e3319-9c8a-480e-aa41-4e7a98c0532a-tmp-dir\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.909507 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.909468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57prl\" (UniqueName: \"kubernetes.io/projected/a8d42714-2762-4696-8f20-c1005f745797-kube-api-access-57prl\") pod \"ingress-canary-5w4jn\" (UID: \"a8d42714-2762-4696-8f20-c1005f745797\") " pod="openshift-ingress-canary/ingress-canary-5w4jn"
Aug 01 00:34:28.909994 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.909615    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d36e3319-9c8a-480e-aa41-4e7a98c0532a-config-volume\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.909994 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.909658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8d42714-2762-4696-8f20-c1005f745797-cert\") pod \"ingress-canary-5w4jn\" (UID: \"a8d42714-2762-4696-8f20-c1005f745797\") " pod="openshift-ingress-canary/ingress-canary-5w4jn"
Aug 01 00:34:28.909994 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.909685    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.909994 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.909717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdsjc\" (UniqueName: \"kubernetes.io/projected/d36e3319-9c8a-480e-aa41-4e7a98c0532a-kube-api-access-xdsjc\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.909994 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.909819    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d36e3319-9c8a-480e-aa41-4e7a98c0532a-tmp-dir\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.909994 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:28.909847    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:34:28.909994 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:28.909890    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:34:29.409877355 +0000 UTC m=+35.643809599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:34:28.910278 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.910258    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d36e3319-9c8a-480e-aa41-4e7a98c0532a-config-volume\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.914844 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.914825    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8d42714-2762-4696-8f20-c1005f745797-cert\") pod \"ingress-canary-5w4jn\" (UID: \"a8d42714-2762-4696-8f20-c1005f745797\") " pod="openshift-ingress-canary/ingress-canary-5w4jn"
Aug 01 00:34:28.918350 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.918329    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57prl\" (UniqueName: \"kubernetes.io/projected/a8d42714-2762-4696-8f20-c1005f745797-kube-api-access-57prl\") pod \"ingress-canary-5w4jn\" (UID: \"a8d42714-2762-4696-8f20-c1005f745797\") " pod="openshift-ingress-canary/ingress-canary-5w4jn"
Aug 01 00:34:28.918442 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.918410    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdsjc\" (UniqueName: \"kubernetes.io/projected/d36e3319-9c8a-480e-aa41-4e7a98c0532a-kube-api-access-xdsjc\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:28.975544 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:28.975523    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5w4jn"
Aug 01 00:34:29.115028 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.114846    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5w4jn"]
Aug 01 00:34:29.118352 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:34:29.118330    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d42714_2762_4696_8f20_c1005f745797.slice/crio-dac63815903d8291c93c9a4d7337865017a84e7bbcbe70aa2e7b43c1c9f8ba68 WatchSource:0}: Error finding container dac63815903d8291c93c9a4d7337865017a84e7bbcbe70aa2e7b43c1c9f8ba68: Status 404 returned error can't find the container with id dac63815903d8291c93c9a4d7337865017a84e7bbcbe70aa2e7b43c1c9f8ba68
Aug 01 00:34:29.256498 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.256434    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:34:29.256616 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.256434    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:29.256616 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.256434    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:29.259617 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.259550    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 01 00:34:29.259617 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.259642    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 01 00:34:29.259857 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.259644    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hdfd9\""
Aug 01 00:34:29.260320 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.260299    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 01 00:34:29.260320 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.260315    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 01 00:34:29.260489 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.260299    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hk5qd\""
Aug 01 00:34:29.414451 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.414422    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:29.414614 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:29.414590    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:34:29.414732 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:29.414712    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:34:30.414689756 +0000 UTC m=+36.648621999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:34:29.438552 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:29.438518    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5w4jn" event={"ID":"a8d42714-2762-4696-8f20-c1005f745797","Type":"ContainerStarted","Data":"dac63815903d8291c93c9a4d7337865017a84e7bbcbe70aa2e7b43c1c9f8ba68"}
Aug 01 00:34:30.421468 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:30.421387    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:30.421890 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:30.421522    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:34:30.421890 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:30.421582    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:34:32.421568235 +0000 UTC m=+38.655500480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:34:31.446391 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:31.446355    2570 generic.go:356] "Generic (PLEG): container finished" podID="c2ff6e14-3a7a-4d02-a3c1-5d4e34712996" containerID="a7cc4547a96381b2ddf3b6d521ad3cd1eb7620aacce47fcb0cd4cfc5c357a023" exitCode=0
Aug 01 00:34:31.446790 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:31.446411    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" event={"ID":"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996","Type":"ContainerDied","Data":"a7cc4547a96381b2ddf3b6d521ad3cd1eb7620aacce47fcb0cd4cfc5c357a023"}
Aug 01 00:34:32.437070 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:32.436844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:32.437243 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:32.436994    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:34:32.437243 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:32.437206    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:34:36.437185714 +0000 UTC m=+42.671117957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:34:32.455358 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:32.455321    2570 generic.go:356] "Generic (PLEG): container finished" podID="c2ff6e14-3a7a-4d02-a3c1-5d4e34712996" containerID="6d57b3772a939c975d1a735f76fa41af76ee6867d02c8dcfeb581dd83821558d" exitCode=0
Aug 01 00:34:32.455739 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:32.455369    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" event={"ID":"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996","Type":"ContainerDied","Data":"6d57b3772a939c975d1a735f76fa41af76ee6867d02c8dcfeb581dd83821558d"}
Aug 01 00:34:33.460625 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:33.460543    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" event={"ID":"c2ff6e14-3a7a-4d02-a3c1-5d4e34712996","Type":"ContainerStarted","Data":"59557415558fa2b2247d0159d9e4e79d671d54a9db2c24d1fc452f8a37792991"}
Aug 01 00:34:33.462025 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:33.462002    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5w4jn" event={"ID":"a8d42714-2762-4696-8f20-c1005f745797","Type":"ContainerStarted","Data":"5e73af2d45be9b248c34fd5cf24ac23839aa1eff1201047a57916a6761724fd2"}
Aug 01 00:34:33.487091 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:33.487050    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tq6z2" podStartSLOduration=9.218359572 podStartE2EDuration="39.487035626s" totalImagesPullingTime="30.268676054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:33.485278589 +0000 UTC m=+39.719210853" watchObservedRunningTime="2026-08-01 00:34:33.487035626 +0000 UTC m=+39.720967885"
Aug 01 00:34:33.501733 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:33.501690    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5w4jn" podStartSLOduration=1.5248369290000001 podStartE2EDuration="5.50167431s" totalImagesPullingTime="3.976837381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:33.501231558 +0000 UTC m=+39.735163823" watchObservedRunningTime="2026-08-01 00:34:33.50167431 +0000 UTC m=+39.735606576"
Aug 01 00:34:36.462978 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:36.462940    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:36.463405 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:36.463040    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:34:36.463405 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:36.463122    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:34:44.463097638 +0000 UTC m=+50.697029881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:34:44.514133 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:44.514101    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:34:44.514565 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:44.514219    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:34:44.514565 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:34:44.514273    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:35:00.514260247 +0000 UTC m=+66.748192491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:34:54.432014 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:54.431978    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qd5f8"
Aug 01 00:34:59.919361 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:59.919306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:59.919361 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:59.919362    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:34:59.922168 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:59.922147    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 01 00:34:59.922168 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:59.922163    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 01 00:34:59.932156 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:59.932138    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 01 00:34:59.933777 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:59.933756    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f5c55ca5-8428-4fbf-acbf-5c3d996c68d5-original-pull-secret\") pod \"global-pull-secret-syncer-pgtlh\" (UID: \"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5\") " pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:34:59.942963 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:34:59.942925    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lz9\" (UniqueName: \"kubernetes.io/projected/1c57de4c-d851-4099-858a-550058fabda9-kube-api-access-s2lz9\") pod \"network-check-target-xxpkk\" (UID: \"1c57de4c-d851-4099-858a-550058fabda9\") " pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:35:00.020010 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.019979    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:35:00.022380 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.022365    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 01 00:35:00.030343 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:35:00.030317    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 01 00:35:00.030413 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:35:00.030388    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:36:04.030373872 +0000 UTC m=+130.264306115 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : secret "metrics-daemon-secret" not found
Aug 01 00:35:00.179586 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.179493    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hk5qd\""
Aug 01 00:35:00.181715 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.181700    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pgtlh"
Aug 01 00:35:00.186847 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.186828    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:35:00.309131 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.309048    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pgtlh"]
Aug 01 00:35:00.312822 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:35:00.312797    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c55ca5_8428_4fbf_acbf_5c3d996c68d5.slice/crio-6331201d7bcd81008343fc0f144d157d4f472eeb4eb437240a1e9283a1fe1226 WatchSource:0}: Error finding container 6331201d7bcd81008343fc0f144d157d4f472eeb4eb437240a1e9283a1fe1226: Status 404 returned error can't find the container with id 6331201d7bcd81008343fc0f144d157d4f472eeb4eb437240a1e9283a1fe1226
Aug 01 00:35:00.324336 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.324305    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xxpkk"]
Aug 01 00:35:00.326804 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:35:00.326784    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c57de4c_d851_4099_858a_550058fabda9.slice/crio-d84bec9ac8cfa9f2f0c8eb1148ca576dcee28523f4870238a89ee0310fd6820c WatchSource:0}: Error finding container d84bec9ac8cfa9f2f0c8eb1148ca576dcee28523f4870238a89ee0310fd6820c: Status 404 returned error can't find the container with id d84bec9ac8cfa9f2f0c8eb1148ca576dcee28523f4870238a89ee0310fd6820c
Aug 01 00:35:00.522192 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.522103    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:35:00.522365 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:35:00.522214    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:35:00.522365 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:35:00.522267    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:35:32.522252791 +0000 UTC m=+98.756185035 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:35:00.522505 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.522486    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xxpkk" event={"ID":"1c57de4c-d851-4099-858a-550058fabda9","Type":"ContainerStarted","Data":"d84bec9ac8cfa9f2f0c8eb1148ca576dcee28523f4870238a89ee0310fd6820c"}
Aug 01 00:35:00.523466 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:00.523443    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pgtlh" event={"ID":"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5","Type":"ContainerStarted","Data":"6331201d7bcd81008343fc0f144d157d4f472eeb4eb437240a1e9283a1fe1226"}
Aug 01 00:35:05.540056 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:05.540015    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xxpkk" event={"ID":"1c57de4c-d851-4099-858a-550058fabda9","Type":"ContainerStarted","Data":"a18ddba6454b9fa1523b93fbeb804b1a1acbaaca5955429bc421faab76f2669a"}
Aug 01 00:35:05.540495 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:05.540151    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:35:05.541399 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:05.541374    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pgtlh" event={"ID":"f5c55ca5-8428-4fbf-acbf-5c3d996c68d5","Type":"ContainerStarted","Data":"8240e833b223900b46d6db45ad62c57aecd4e6aef2bb8f0c971422af4b574a13"}
Aug 01 00:35:05.555158 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:05.555117    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xxpkk" podStartSLOduration=66.984755878 podStartE2EDuration="1m11.555106003s" totalImagesPullingTime="4.570350125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:35:05.554240698 +0000 UTC m=+71.788172963" watchObservedRunningTime="2026-08-01 00:35:05.555106003 +0000 UTC m=+71.789038268"
Aug 01 00:35:05.567907 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:05.567863    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pgtlh" podStartSLOduration=66.978857102 podStartE2EDuration="1m11.567851625s" totalImagesPullingTime="4.588994523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:35:05.567454971 +0000 UTC m=+71.801387236" watchObservedRunningTime="2026-08-01 00:35:05.567851625 +0000 UTC m=+71.801783891"
Aug 01 00:35:32.532961 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:32.532799    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:35:32.532961 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:35:32.532963    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:35:32.533493 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:35:32.533030    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:36:36.533012016 +0000 UTC m=+162.766944263 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:35:36.546028 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:35:36.545998    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xxpkk"
Aug 01 00:36:04.034982 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:04.034926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:36:04.035550 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:36:04.035101    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 01 00:36:04.035550 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:36:04.035207    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs podName:31926079-d63f-43de-b52c-ced2e283ba10 nodeName:}" failed. No retries permitted until 2026-08-01 00:38:06.035182909 +0000 UTC m=+252.269115154 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs") pod "network-metrics-daemon-99f8t" (UID: "31926079-d63f-43de-b52c-ced2e283ba10") : secret "metrics-daemon-secret" not found
Aug 01 00:36:20.957212 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.957169    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"]
Aug 01 00:36:20.958982 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.958955    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"
Aug 01 00:36:20.961443 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.961424    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Aug 01 00:36:20.962809 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.962777    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 01 00:36:20.962809 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.962791    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 01 00:36:20.962809 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.962801    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 01 00:36:20.963009 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.962853    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pc8vj\""
Aug 01 00:36:20.967057 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.967033    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"]
Aug 01 00:36:20.978624 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.978601    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"]
Aug 01 00:36:20.980430 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.980414    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:20.983029 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.983011    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Aug 01 00:36:20.990019 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:20.989998    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"]
Aug 01 00:36:21.039519 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.039483    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqknv\" (UniqueName: \"kubernetes.io/projected/cf1a71e4-e34e-4959-a913-91deeb31bda8-kube-api-access-tqknv\") pod \"managed-serviceaccount-addon-agent-5f6657556-q8n94\" (UID: \"cf1a71e4-e34e-4959-a913-91deeb31bda8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"
Aug 01 00:36:21.039708 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.039555    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf1a71e4-e34e-4959-a913-91deeb31bda8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f6657556-q8n94\" (UID: \"cf1a71e4-e34e-4959-a913-91deeb31bda8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"
Aug 01 00:36:21.140245 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.140207    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz6mc\" (UniqueName: \"kubernetes.io/projected/a828bf62-04ca-40fa-928e-ea7080a7a1e7-kube-api-access-cz6mc\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.140245 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.140250    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqknv\" (UniqueName: \"kubernetes.io/projected/cf1a71e4-e34e-4959-a913-91deeb31bda8-kube-api-access-tqknv\") pod \"managed-serviceaccount-addon-agent-5f6657556-q8n94\" (UID: \"cf1a71e4-e34e-4959-a913-91deeb31bda8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"
Aug 01 00:36:21.140472 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.140387    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf1a71e4-e34e-4959-a913-91deeb31bda8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f6657556-q8n94\" (UID: \"cf1a71e4-e34e-4959-a913-91deeb31bda8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"
Aug 01 00:36:21.140472 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.140443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a828bf62-04ca-40fa-928e-ea7080a7a1e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.140572 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.140487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a828bf62-04ca-40fa-928e-ea7080a7a1e7-tmp\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.142805 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.142782    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf1a71e4-e34e-4959-a913-91deeb31bda8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f6657556-q8n94\" (UID: \"cf1a71e4-e34e-4959-a913-91deeb31bda8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"
Aug 01 00:36:21.148347 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.148328    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqknv\" (UniqueName: \"kubernetes.io/projected/cf1a71e4-e34e-4959-a913-91deeb31bda8-kube-api-access-tqknv\") pod \"managed-serviceaccount-addon-agent-5f6657556-q8n94\" (UID: \"cf1a71e4-e34e-4959-a913-91deeb31bda8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"
Aug 01 00:36:21.241437 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.241346    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a828bf62-04ca-40fa-928e-ea7080a7a1e7-tmp\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.241437 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.241390    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz6mc\" (UniqueName: \"kubernetes.io/projected/a828bf62-04ca-40fa-928e-ea7080a7a1e7-kube-api-access-cz6mc\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.241619 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.241442    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a828bf62-04ca-40fa-928e-ea7080a7a1e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.241761 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.241731    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a828bf62-04ca-40fa-928e-ea7080a7a1e7-tmp\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.243801 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.243773    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a828bf62-04ca-40fa-928e-ea7080a7a1e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.249461 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.249438    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz6mc\" (UniqueName: \"kubernetes.io/projected/a828bf62-04ca-40fa-928e-ea7080a7a1e7-kube-api-access-cz6mc\") pod \"klusterlet-addon-workmgr-654685d4b5-75pqg\" (UID: \"a828bf62-04ca-40fa-928e-ea7080a7a1e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.278765 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.278730    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"
Aug 01 00:36:21.288759 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.288730    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:21.405886 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.405855    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94"]
Aug 01 00:36:21.409057 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:36:21.409032    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf1a71e4_e34e_4959_a913_91deeb31bda8.slice/crio-1367eb21f61e606cc24c3bb1700ea03bc19101b03e2277f3f98bddccb5a1a5bd WatchSource:0}: Error finding container 1367eb21f61e606cc24c3bb1700ea03bc19101b03e2277f3f98bddccb5a1a5bd: Status 404 returned error can't find the container with id 1367eb21f61e606cc24c3bb1700ea03bc19101b03e2277f3f98bddccb5a1a5bd
Aug 01 00:36:21.418647 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.418624    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"]
Aug 01 00:36:21.421702 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:36:21.421679    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda828bf62_04ca_40fa_928e_ea7080a7a1e7.slice/crio-ae45a5e0ddfaed810da83b1ec1ce403fffeb47344635a2987cbfb7f72d9e907f WatchSource:0}: Error finding container ae45a5e0ddfaed810da83b1ec1ce403fffeb47344635a2987cbfb7f72d9e907f: Status 404 returned error can't find the container with id ae45a5e0ddfaed810da83b1ec1ce403fffeb47344635a2987cbfb7f72d9e907f
Aug 01 00:36:21.720528 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.720490    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg" event={"ID":"a828bf62-04ca-40fa-928e-ea7080a7a1e7","Type":"ContainerStarted","Data":"ae45a5e0ddfaed810da83b1ec1ce403fffeb47344635a2987cbfb7f72d9e907f"}
Aug 01 00:36:21.721560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:21.721535    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94" event={"ID":"cf1a71e4-e34e-4959-a913-91deeb31bda8","Type":"ContainerStarted","Data":"1367eb21f61e606cc24c3bb1700ea03bc19101b03e2277f3f98bddccb5a1a5bd"}
Aug 01 00:36:25.733742 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:25.733666    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94" event={"ID":"cf1a71e4-e34e-4959-a913-91deeb31bda8","Type":"ContainerStarted","Data":"4ee949dab7798fd66322d2b4d7f0f48dfa18620d900fea8567ed0870838503cc"}
Aug 01 00:36:25.749063 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:25.749018    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94" podStartSLOduration=1.7431527089999999 podStartE2EDuration="5.749002611s" totalImagesPullingTime="4.005849902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:36:25.748825613 +0000 UTC m=+151.982757883" watchObservedRunningTime="2026-08-01 00:36:25.749002611 +0000 UTC m=+151.982934877"
Aug 01 00:36:27.740282 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:27.740206    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg" event={"ID":"a828bf62-04ca-40fa-928e-ea7080a7a1e7","Type":"ContainerStarted","Data":"1ee577a69da65c1b5f71bb6cbb0f922a0c817001c25a3eb7a689e5faf291bcc9"}
Aug 01 00:36:27.740624 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:27.740446    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:27.742072 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:27.742003    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:36:27.756984 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:27.756920    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg" podStartSLOduration=1.719401767 podStartE2EDuration="7.756908392s" totalImagesPullingTime="6.037506625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:36:27.755349694 +0000 UTC m=+153.989281951" watchObservedRunningTime="2026-08-01 00:36:27.756908392 +0000 UTC m=+153.990840685"
Aug 01 00:36:31.669979 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:36:31.669905    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-r7rdc" podUID="d36e3319-9c8a-480e-aa41-4e7a98c0532a"
Aug 01 00:36:31.749993 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:31.749962    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:36:32.268754 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:36:32.268716    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-99f8t" podUID="31926079-d63f-43de-b52c-ced2e283ba10"
Aug 01 00:36:36.541515 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:36.541461    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:36:36.542062 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:36:36.541618    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 01 00:36:36.542062 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:36:36.541694    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls podName:d36e3319-9c8a-480e-aa41-4e7a98c0532a nodeName:}" failed. No retries permitted until 2026-08-01 00:38:38.541677613 +0000 UTC m=+284.775609860 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls") pod "dns-default-r7rdc" (UID: "d36e3319-9c8a-480e-aa41-4e7a98c0532a") : secret "dns-default-metrics-tls" not found
Aug 01 00:36:43.256580 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:36:43.256537    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:37:38.710081 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.710046    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56c649478c-nzscm"]
Aug 01 00:37:38.712914 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.712897    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.715874 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.715854    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4zgh7\""
Aug 01 00:37:38.716145 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.716131    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 01 00:37:38.716470 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.716455    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 01 00:37:38.719088 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.719070    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 01 00:37:38.724699 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.724679    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 01 00:37:38.747438 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.747416    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56c649478c-nzscm"]
Aug 01 00:37:38.767028 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.767006    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-trusted-ca\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.767128 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.767031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-installation-pull-secrets\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.767128 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.767063    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-bound-sa-token\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.767128 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.767110    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-image-registry-private-configuration\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.767334 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.767177    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvprn\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-kube-api-access-lvprn\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.767334 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.767204    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-registry-certificates\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.767334 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.767245    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.767334 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.767267    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f014511-d183-4684-9e4d-3fb406908b03-ca-trust-extracted\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868243 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868213    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-registry-certificates\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868408 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868253    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868408 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868282    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f014511-d183-4684-9e4d-3fb406908b03-ca-trust-extracted\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868408 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-trusted-ca\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868408 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868340    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-installation-pull-secrets\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868408 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868382    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-bound-sa-token\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868408 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:38.868391    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 01 00:37:38.868408 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868407    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-image-registry-private-configuration\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868751 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:38.868430    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c649478c-nzscm: secret "image-registry-tls" not found
Aug 01 00:37:38.868751 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:38.868531    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls podName:1f014511-d183-4684-9e4d-3fb406908b03 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:39.368506399 +0000 UTC m=+225.602438667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls") pod "image-registry-56c649478c-nzscm" (UID: "1f014511-d183-4684-9e4d-3fb406908b03") : secret "image-registry-tls" not found
Aug 01 00:37:38.868751 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868609    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvprn\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-kube-api-access-lvprn\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868905 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868796    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f014511-d183-4684-9e4d-3fb406908b03-ca-trust-extracted\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.868980 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.868904    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-registry-certificates\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.869237 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.869218    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-trusted-ca\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.870767 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.870747    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-installation-pull-secrets\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.870818 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.870807    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-image-registry-private-configuration\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.878383 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.878363    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvprn\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-kube-api-access-lvprn\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:38.879633 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:38.879617    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-bound-sa-token\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:39.373517 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:39.373473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:39.373700 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:39.373627    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 01 00:37:39.373700 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:39.373649    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c649478c-nzscm: secret "image-registry-tls" not found
Aug 01 00:37:39.373784 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:39.373733    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls podName:1f014511-d183-4684-9e4d-3fb406908b03 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:40.373717128 +0000 UTC m=+226.607649373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls") pod "image-registry-56c649478c-nzscm" (UID: "1f014511-d183-4684-9e4d-3fb406908b03") : secret "image-registry-tls" not found
Aug 01 00:37:40.380008 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:40.379972    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:40.380394 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:40.380093    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 01 00:37:40.380394 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:40.380106    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c649478c-nzscm: secret "image-registry-tls" not found
Aug 01 00:37:40.380394 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:40.380163    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls podName:1f014511-d183-4684-9e4d-3fb406908b03 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:42.380148201 +0000 UTC m=+228.614080445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls") pod "image-registry-56c649478c-nzscm" (UID: "1f014511-d183-4684-9e4d-3fb406908b03") : secret "image-registry-tls" not found
Aug 01 00:37:42.394235 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:42.394188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:42.394630 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:42.394313    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 01 00:37:42.394630 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:42.394325    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c649478c-nzscm: secret "image-registry-tls" not found
Aug 01 00:37:42.394630 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:42.394374    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls podName:1f014511-d183-4684-9e4d-3fb406908b03 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:46.394361139 +0000 UTC m=+232.628293383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls") pod "image-registry-56c649478c-nzscm" (UID: "1f014511-d183-4684-9e4d-3fb406908b03") : secret "image-registry-tls" not found
Aug 01 00:37:44.060720 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.060688    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962"]
Aug 01 00:37:44.064260 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.064244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962"
Aug 01 00:37:44.066877 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.066861    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Aug 01 00:37:44.066981 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.066889    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nx2s9\""
Aug 01 00:37:44.066981 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.066970    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Aug 01 00:37:44.073593 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.073569    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962"]
Aug 01 00:37:44.211381 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.208535    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddn6\" (UniqueName: \"kubernetes.io/projected/bb9510cb-97a2-4d67-9ffe-0d0902e6bf04-kube-api-access-mddn6\") pod \"migrator-68bc745d87-ck962\" (UID: \"bb9510cb-97a2-4d67-9ffe-0d0902e6bf04\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962"
Aug 01 00:37:44.270600 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.270570    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q4qrs"]
Aug 01 00:37:44.273560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.273541    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.276333 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.276309    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jls8z\""
Aug 01 00:37:44.276429 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.276339    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 01 00:37:44.276429 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.276350    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 01 00:37:44.276843 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.276827    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 01 00:37:44.276843 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.276836    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 01 00:37:44.285290 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.285273    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4qrs"]
Aug 01 00:37:44.309452 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.309421    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddn6\" (UniqueName: \"kubernetes.io/projected/bb9510cb-97a2-4d67-9ffe-0d0902e6bf04-kube-api-access-mddn6\") pod \"migrator-68bc745d87-ck962\" (UID: \"bb9510cb-97a2-4d67-9ffe-0d0902e6bf04\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962"
Aug 01 00:37:44.317196 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.317176    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddn6\" (UniqueName: \"kubernetes.io/projected/bb9510cb-97a2-4d67-9ffe-0d0902e6bf04-kube-api-access-mddn6\") pod \"migrator-68bc745d87-ck962\" (UID: \"bb9510cb-97a2-4d67-9ffe-0d0902e6bf04\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962"
Aug 01 00:37:44.372537 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.372495    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962"
Aug 01 00:37:44.410666 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.410634    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-data-volume\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.410827 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.410669    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.410827 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.410711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.410827 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.410733    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj544\" (UniqueName: \"kubernetes.io/projected/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-api-access-jj544\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.410827 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.410791    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-crio-socket\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.492866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.492829    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962"]
Aug 01 00:37:44.495288 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:37:44.495263    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9510cb_97a2_4d67_9ffe_0d0902e6bf04.slice/crio-82fa6a6909d072905aa8f45ee5bdc4d5fe663ca12237713be805b958803ab8c3 WatchSource:0}: Error finding container 82fa6a6909d072905aa8f45ee5bdc4d5fe663ca12237713be805b958803ab8c3: Status 404 returned error can't find the container with id 82fa6a6909d072905aa8f45ee5bdc4d5fe663ca12237713be805b958803ab8c3
Aug 01 00:37:44.511502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.511480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-crio-socket\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.511591 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.511529    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-data-volume\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.511644 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.511600    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-crio-socket\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.511697 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.511637    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.511697 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.511694    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.511792 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.511711    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj544\" (UniqueName: \"kubernetes.io/projected/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-api-access-jj544\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.511792 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.511751    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-data-volume\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.511885 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:44.511794    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 01 00:37:44.511885 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:44.511804    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:44.511885 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:44.511858    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:45.011843065 +0000 UTC m=+231.245775309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : configmap "kube-rbac-proxy" not found
Aug 01 00:37:44.511885 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:44.511874    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:45.011866495 +0000 UTC m=+231.245798741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:44.520036 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.520020    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj544\" (UniqueName: \"kubernetes.io/projected/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-api-access-jj544\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:44.932128 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:44.932091    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962" event={"ID":"bb9510cb-97a2-4d67-9ffe-0d0902e6bf04","Type":"ContainerStarted","Data":"82fa6a6909d072905aa8f45ee5bdc4d5fe663ca12237713be805b958803ab8c3"}
Aug 01 00:37:45.015752 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:45.015708    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:45.015954 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:45.015838    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:45.015954 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:45.015840    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 01 00:37:45.016036 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:45.015955    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:46.015919577 +0000 UTC m=+232.249851840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : configmap "kube-rbac-proxy" not found
Aug 01 00:37:45.016036 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:45.015955    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:45.016036 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:45.015989    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:46.015980921 +0000 UTC m=+232.249913165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:45.936477 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:45.936395    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962" event={"ID":"bb9510cb-97a2-4d67-9ffe-0d0902e6bf04","Type":"ContainerStarted","Data":"ed8de20a12ac8b0999b3ea54c8604147e2a96aba5722afcea64195b4db231655"}
Aug 01 00:37:45.936477 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:45.936445    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962" event={"ID":"bb9510cb-97a2-4d67-9ffe-0d0902e6bf04","Type":"ContainerStarted","Data":"ab8805f21b0a96f2982bb5b45703fc325e2aaa863abc71316a6505067a7f98dd"}
Aug 01 00:37:45.956207 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:45.954105    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-ck962" podStartSLOduration=0.792471349 podStartE2EDuration="1.954085384s" totalImagesPullingTime="1.161614035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:45.952179249 +0000 UTC m=+232.186111514" watchObservedRunningTime="2026-08-01 00:37:45.954085384 +0000 UTC m=+232.188017653"
Aug 01 00:37:46.023200 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:46.023169    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:46.023368 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:46.023214    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:46.023368 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:46.023275    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 01 00:37:46.023368 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:46.023315    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:46.023368 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:46.023322    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:48.023310095 +0000 UTC m=+234.257242343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : configmap "kube-rbac-proxy" not found
Aug 01 00:37:46.023535 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:46.023388    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:48.023370529 +0000 UTC m=+234.257302773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:46.426737 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:46.426681    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:46.426919 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:46.426839    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 01 00:37:46.426919 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:46.426860    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c649478c-nzscm: secret "image-registry-tls" not found
Aug 01 00:37:46.427115 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:46.426922    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls podName:1f014511-d183-4684-9e4d-3fb406908b03 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:54.426903612 +0000 UTC m=+240.660835875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls") pod "image-registry-56c649478c-nzscm" (UID: "1f014511-d183-4684-9e4d-3fb406908b03") : secret "image-registry-tls" not found
Aug 01 00:37:46.837522 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:46.837500    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k8jhp_ad989299-a180-4b5d-ae39-9a5ffc294368/dns-node-resolver/0.log"
Aug 01 00:37:48.037437 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:48.037410    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5fvlj_85d6afc8-b208-4fcf-b770-47f6560962a3/node-ca/0.log"
Aug 01 00:37:48.038587 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:48.038567    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:48.038659 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:48.038608    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:48.038705 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:48.038694    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:48.038747 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:48.038711    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 01 00:37:48.038791 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:48.038747    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:52.038731099 +0000 UTC m=+238.272663343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:48.038791 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:48.038761    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:52.038754797 +0000 UTC m=+238.272687041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : configmap "kube-rbac-proxy" not found
Aug 01 00:37:48.637752 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:48.637723    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5w4jn_a8d42714-2762-4696-8f20-c1005f745797/serve-healthcheck-canary/0.log"
Aug 01 00:37:49.237422 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:49.237392    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-ck962_bb9510cb-97a2-4d67-9ffe-0d0902e6bf04/migrator/0.log"
Aug 01 00:37:49.437552 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:49.437524    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-ck962_bb9510cb-97a2-4d67-9ffe-0d0902e6bf04/graceful-termination/0.log"
Aug 01 00:37:52.067336 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:52.067297    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:52.067336 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:52.067351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:37:52.067761 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:52.067463    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 01 00:37:52.067761 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:52.067467    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:52.067761 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:52.067519    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:38:00.067503272 +0000 UTC m=+246.301435521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : configmap "kube-rbac-proxy" not found
Aug 01 00:37:52.067761 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:37:52.067534    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls podName:c91d0d2b-a7e4-4d7c-b252-86e77bd86964 nodeName:}" failed. No retries permitted until 2026-08-01 00:38:00.067527576 +0000 UTC m=+246.301459820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4qrs" (UID: "c91d0d2b-a7e4-4d7c-b252-86e77bd86964") : secret "insights-runtime-extractor-tls" not found
Aug 01 00:37:54.490256 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.490226    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:54.492454 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.492429    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"image-registry-56c649478c-nzscm\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") " pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:54.624242 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.624220    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4zgh7\""
Aug 01 00:37:54.632392 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.632375    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:54.756764 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.756699    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56c649478c-nzscm"]
Aug 01 00:37:54.760092 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:37:54.760068    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f014511_d183_4684_9e4d_3fb406908b03.slice/crio-3af8c12cc4d13e484af1c2ec5d0721c192cc66fa9367b8d185adc44571a0ee00 WatchSource:0}: Error finding container 3af8c12cc4d13e484af1c2ec5d0721c192cc66fa9367b8d185adc44571a0ee00: Status 404 returned error can't find the container with id 3af8c12cc4d13e484af1c2ec5d0721c192cc66fa9367b8d185adc44571a0ee00
Aug 01 00:37:54.965383 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.965346    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c649478c-nzscm" event={"ID":"1f014511-d183-4684-9e4d-3fb406908b03","Type":"ContainerStarted","Data":"81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af"}
Aug 01 00:37:54.965572 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.965393    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c649478c-nzscm" event={"ID":"1f014511-d183-4684-9e4d-3fb406908b03","Type":"ContainerStarted","Data":"3af8c12cc4d13e484af1c2ec5d0721c192cc66fa9367b8d185adc44571a0ee00"}
Aug 01 00:37:54.965572 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.965484    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:37:54.984846 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:37:54.984686    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56c649478c-nzscm" podStartSLOduration=16.984670663 podStartE2EDuration="16.984670663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:54.984050934 +0000 UTC m=+241.217983199" watchObservedRunningTime="2026-08-01 00:37:54.984670663 +0000 UTC m=+241.218602946"
Aug 01 00:38:00.133484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.133448    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:38:00.133484 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.133489    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:38:00.133992 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.133976    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:38:00.135659 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.135638    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c91d0d2b-a7e4-4d7c-b252-86e77bd86964-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4qrs\" (UID: \"c91d0d2b-a7e4-4d7c-b252-86e77bd86964\") " pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:38:00.184836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.184808    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jls8z\""
Aug 01 00:38:00.192917 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.192893    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4qrs"
Aug 01 00:38:00.316395 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.316364    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4qrs"]
Aug 01 00:38:00.319189 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:00.319161    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91d0d2b_a7e4_4d7c_b252_86e77bd86964.slice/crio-bae187305fa3362edcadba8c15bb45e97c095c3f1dcf286f51cfd047eb136a0e WatchSource:0}: Error finding container bae187305fa3362edcadba8c15bb45e97c095c3f1dcf286f51cfd047eb136a0e: Status 404 returned error can't find the container with id bae187305fa3362edcadba8c15bb45e97c095c3f1dcf286f51cfd047eb136a0e
Aug 01 00:38:00.985547 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.985508    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4qrs" event={"ID":"c91d0d2b-a7e4-4d7c-b252-86e77bd86964","Type":"ContainerStarted","Data":"8317c9b2131de9a4a928b706ef67ec0029ce3ae711725019acb075aaaaeadf3f"}
Aug 01 00:38:00.985744 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:00.985581    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4qrs" event={"ID":"c91d0d2b-a7e4-4d7c-b252-86e77bd86964","Type":"ContainerStarted","Data":"bae187305fa3362edcadba8c15bb45e97c095c3f1dcf286f51cfd047eb136a0e"}
Aug 01 00:38:01.990427 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:01.990395    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4qrs" event={"ID":"c91d0d2b-a7e4-4d7c-b252-86e77bd86964","Type":"ContainerStarted","Data":"a5cc96a402827be2c4ed501e42300259594f72382a2b1810606e151b5302cecd"}
Aug 01 00:38:02.995974 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:02.995873    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4qrs" event={"ID":"c91d0d2b-a7e4-4d7c-b252-86e77bd86964","Type":"ContainerStarted","Data":"f0651d5b12a44d0f6c76bfed82af4ce87d43b519e9167e2858f31331a1a78ebc"}
Aug 01 00:38:04.835103 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:04.835040    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q4qrs" podStartSLOduration=18.577822793 podStartE2EDuration="20.835020868s" totalImagesPullingTime="2.257198075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:37:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:03.013248019 +0000 UTC m=+249.247180296" watchObservedRunningTime="2026-08-01 00:38:04.835020868 +0000 UTC m=+251.068953135"
Aug 01 00:38:04.835976 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:04.835956    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56c649478c-nzscm"]
Aug 01 00:38:04.858449 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:04.858424    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"]
Aug 01 00:38:04.860864 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:04.860850    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"
Aug 01 00:38:04.863421 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:04.863402    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Aug 01 00:38:04.863523 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:04.863402    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6mbqq\""
Aug 01 00:38:04.872346 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:04.872323    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"]
Aug 01 00:38:04.966977 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:04.966946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eeb95623-3f3a-40e6-8145-ddecda8ff61a-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-66f5h\" (UID: \"eeb95623-3f3a-40e6-8145-ddecda8ff61a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"
Aug 01 00:38:05.067746 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:05.067722    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eeb95623-3f3a-40e6-8145-ddecda8ff61a-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-66f5h\" (UID: \"eeb95623-3f3a-40e6-8145-ddecda8ff61a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"
Aug 01 00:38:05.070129 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:05.070099    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eeb95623-3f3a-40e6-8145-ddecda8ff61a-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-66f5h\" (UID: \"eeb95623-3f3a-40e6-8145-ddecda8ff61a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"
Aug 01 00:38:05.169859 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:05.169802    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"
Aug 01 00:38:05.288242 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:05.288221    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"]
Aug 01 00:38:05.290589 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:05.290564    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb95623_3f3a_40e6_8145_ddecda8ff61a.slice/crio-31e13224fa65168a4ce2e84a3f94e8baca31916f29f94f910f01360761891f50 WatchSource:0}: Error finding container 31e13224fa65168a4ce2e84a3f94e8baca31916f29f94f910f01360761891f50: Status 404 returned error can't find the container with id 31e13224fa65168a4ce2e84a3f94e8baca31916f29f94f910f01360761891f50
Aug 01 00:38:06.010709 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:06.010674    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h" event={"ID":"eeb95623-3f3a-40e6-8145-ddecda8ff61a","Type":"ContainerStarted","Data":"31e13224fa65168a4ce2e84a3f94e8baca31916f29f94f910f01360761891f50"}
Aug 01 00:38:06.075651 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:06.075614    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:38:06.078320 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:06.078295    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31926079-d63f-43de-b52c-ced2e283ba10-metrics-certs\") pod \"network-metrics-daemon-99f8t\" (UID: \"31926079-d63f-43de-b52c-ced2e283ba10\") " pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:38:06.359963 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:06.359922    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hdfd9\""
Aug 01 00:38:06.367814 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:06.367801    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-99f8t"
Aug 01 00:38:06.493218 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:06.493191    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-99f8t"]
Aug 01 00:38:06.495996 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:06.495963    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31926079_d63f_43de_b52c_ced2e283ba10.slice/crio-1e4d426e972dd29f5d170eed7ec568ce56bfc1c37eea143bd93ed519f4e83ff6 WatchSource:0}: Error finding container 1e4d426e972dd29f5d170eed7ec568ce56bfc1c37eea143bd93ed519f4e83ff6: Status 404 returned error can't find the container with id 1e4d426e972dd29f5d170eed7ec568ce56bfc1c37eea143bd93ed519f4e83ff6
Aug 01 00:38:07.015835 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.015803    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h" event={"ID":"eeb95623-3f3a-40e6-8145-ddecda8ff61a","Type":"ContainerStarted","Data":"f5fd54dd5a7277cfa9363e862a15da924da17af7d098a8ea3b0026491a8f9575"}
Aug 01 00:38:07.016305 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.015987    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"
Aug 01 00:38:07.017307 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.017280    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-99f8t" event={"ID":"31926079-d63f-43de-b52c-ced2e283ba10","Type":"ContainerStarted","Data":"1e4d426e972dd29f5d170eed7ec568ce56bfc1c37eea143bd93ed519f4e83ff6"}
Aug 01 00:38:07.021513 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.021492    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h"
Aug 01 00:38:07.032593 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.032544    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-66f5h" podStartSLOduration=1.948602277 podStartE2EDuration="3.032525001s" totalImagesPullingTime="1.083922724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:07.031808012 +0000 UTC m=+253.265740278" watchObservedRunningTime="2026-08-01 00:38:07.032525001 +0000 UTC m=+253.266457269"
Aug 01 00:38:07.883866 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.883835    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-54k48"]
Aug 01 00:38:07.886795 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.886780    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:07.891196 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.891025    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8cdcx\""
Aug 01 00:38:07.891196 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.891038    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Aug 01 00:38:07.891196 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.891065    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 01 00:38:07.891196 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.891085    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 01 00:38:07.891196 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.891094    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Aug 01 00:38:07.891196 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.891128    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 01 00:38:07.896098 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.896075    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-54k48"]
Aug 01 00:38:07.987022 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.986997    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:07.987139 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.987042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:07.987139 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.987065    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbppk\" (UniqueName: \"kubernetes.io/projected/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-kube-api-access-xbppk\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:07.987217 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:07.987147    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.022255 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.022220    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-99f8t" event={"ID":"31926079-d63f-43de-b52c-ced2e283ba10","Type":"ContainerStarted","Data":"957aec38edb35d252d4db6659d287ea757d54f015a6a4f69e6146c0b02ba240e"}
Aug 01 00:38:08.022578 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.022281    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-99f8t" event={"ID":"31926079-d63f-43de-b52c-ced2e283ba10","Type":"ContainerStarted","Data":"2924f66b5fb502546c02a5f87a76b437fab616123f6fc03c268d78fa042d86a0"}
Aug 01 00:38:08.037861 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.037822    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-99f8t" podStartSLOduration=253.105307219 podStartE2EDuration="4m14.037810242s" totalImagesPullingTime="932.503023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:08.037635169 +0000 UTC m=+254.271567437" watchObservedRunningTime="2026-08-01 00:38:08.037810242 +0000 UTC m=+254.271742508"
Aug 01 00:38:08.088166 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.088144    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.088253 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.088186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.088253 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.088207    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbppk\" (UniqueName: \"kubernetes.io/projected/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-kube-api-access-xbppk\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.088253 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.088243    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.088860 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.088839    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.090518 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.090493    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.090596 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.090494    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.096037 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.096012    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbppk\" (UniqueName: \"kubernetes.io/projected/a1a483c3-c11e-44e0-9ccf-dbd441ea42fb-kube-api-access-xbppk\") pod \"prometheus-operator-758d7cb874-54k48\" (UID: \"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.195185 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.195137    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48"
Aug 01 00:38:08.315949 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:08.315898    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-54k48"]
Aug 01 00:38:08.319391 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:08.319371    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a483c3_c11e_44e0_9ccf_dbd441ea42fb.slice/crio-ebd6ff4b316d56cbc693ad348054c1711c3fb69e00fb97b7119577df2bf3097f WatchSource:0}: Error finding container ebd6ff4b316d56cbc693ad348054c1711c3fb69e00fb97b7119577df2bf3097f: Status 404 returned error can't find the container with id ebd6ff4b316d56cbc693ad348054c1711c3fb69e00fb97b7119577df2bf3097f
Aug 01 00:38:09.028116 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:09.028079    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48" event={"ID":"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb","Type":"ContainerStarted","Data":"ebd6ff4b316d56cbc693ad348054c1711c3fb69e00fb97b7119577df2bf3097f"}
Aug 01 00:38:10.034851 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:10.034816    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48" event={"ID":"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb","Type":"ContainerStarted","Data":"811fb0d867b91d4f83aa91496d1c036c8d33024f7b4969946bc19455c9ee7b24"}
Aug 01 00:38:10.035258 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:10.034892    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48" event={"ID":"a1a483c3-c11e-44e0-9ccf-dbd441ea42fb","Type":"ContainerStarted","Data":"c47f9104155b93a845a4c57df346b8fae949024e3099b43754374887ae67b09e"}
Aug 01 00:38:10.051408 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:10.051361    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-54k48" podStartSLOduration=1.814543818 podStartE2EDuration="3.051345802s" totalImagesPullingTime="1.236801984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:10.050661145 +0000 UTC m=+256.284593413" watchObservedRunningTime="2026-08-01 00:38:10.051345802 +0000 UTC m=+256.285278067"
Aug 01 00:38:12.214618 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.214583    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"]
Aug 01 00:38:12.217984 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.217962    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.220542 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.220517    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Aug 01 00:38:12.220542 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.220537    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-plp5d\""
Aug 01 00:38:12.220692 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.220527    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Aug 01 00:38:12.229031 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.229011    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"]
Aug 01 00:38:12.242733 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.242706    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t58fj"]
Aug 01 00:38:12.245941 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.245915    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.248377 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.248354    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nvkkd\""
Aug 01 00:38:12.248525 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.248506    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 01 00:38:12.248601 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.248584    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 01 00:38:12.248826 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.248811    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 01 00:38:12.319055 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.319025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.319055 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.319056    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.319266 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.319081    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkb8t\" (UniqueName: \"kubernetes.io/projected/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-kube-api-access-rkb8t\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.319266 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.319148    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.419529 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419489    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d0f282b-d478-4315-b903-74f767be07ae-metrics-client-ca\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419529 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419519    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-sys\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419529 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419538    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-wtmp\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-textfile\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419583    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419623    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-root\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419672    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-tls\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419710    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npzww\" (UniqueName: \"kubernetes.io/projected/8d0f282b-d478-4315-b903-74f767be07ae-kube-api-access-npzww\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419729    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.419824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419749    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkb8t\" (UniqueName: \"kubernetes.io/projected/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-kube-api-access-rkb8t\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.420291 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.419844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.420291 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:38:12.419984    2570 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Aug 01 00:38:12.420291 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:38:12.420044    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-tls podName:bf82be8f-4cfa-4545-aeaf-76d8ecb0d900 nodeName:}" failed. No retries permitted until 2026-08-01 00:38:12.920025446 +0000 UTC m=+259.153957703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-7htgc" (UID: "bf82be8f-4cfa-4545-aeaf-76d8ecb0d900") : secret "openshift-state-metrics-tls" not found
Aug 01 00:38:12.420526 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.420500    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.422093 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.422069    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.430023 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.429996    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkb8t\" (UniqueName: \"kubernetes.io/projected/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-kube-api-access-rkb8t\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.520569 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520493    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d0f282b-d478-4315-b903-74f767be07ae-metrics-client-ca\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520569 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520524    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-sys\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520569 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520544    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-wtmp\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520569 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520565    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-textfile\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520580    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520608    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-root\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520628    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-tls\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520634    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-sys\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520696    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-root\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:38:12.520714    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520725    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npzww\" (UniqueName: \"kubernetes.io/projected/8d0f282b-d478-4315-b903-74f767be07ae-kube-api-access-npzww\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:38:12.520765    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-tls podName:8d0f282b-d478-4315-b903-74f767be07ae nodeName:}" failed. No retries permitted until 2026-08-01 00:38:13.02075221 +0000 UTC m=+259.254684453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-tls") pod "node-exporter-t58fj" (UID: "8d0f282b-d478-4315-b903-74f767be07ae") : secret "node-exporter-tls" not found
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520758    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.520831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.520724    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-wtmp\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.521272 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.521025    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-textfile\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.521272 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.521144    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d0f282b-d478-4315-b903-74f767be07ae-metrics-client-ca\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.521272 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.521261    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.522813 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.522795    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.530577 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.530555    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npzww\" (UniqueName: \"kubernetes.io/projected/8d0f282b-d478-4315-b903-74f767be07ae-kube-api-access-npzww\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:12.924917 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.924887    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:12.927156 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:12.927135    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf82be8f-4cfa-4545-aeaf-76d8ecb0d900-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-7htgc\" (UID: \"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:13.025795 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:13.025768    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-tls\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:13.027836 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:13.027818    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d0f282b-d478-4315-b903-74f767be07ae-node-exporter-tls\") pod \"node-exporter-t58fj\" (UID: \"8d0f282b-d478-4315-b903-74f767be07ae\") " pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:13.127156 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:13.127130    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"
Aug 01 00:38:13.154476 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:13.154443    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t58fj"
Aug 01 00:38:13.164719 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:13.164680    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0f282b_d478_4315_b903_74f767be07ae.slice/crio-098ddc78eeeb714e922a9c699a19042146b44b33709877332d1d89a19104a52e WatchSource:0}: Error finding container 098ddc78eeeb714e922a9c699a19042146b44b33709877332d1d89a19104a52e: Status 404 returned error can't find the container with id 098ddc78eeeb714e922a9c699a19042146b44b33709877332d1d89a19104a52e
Aug 01 00:38:13.258843 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:13.257064    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc"]
Aug 01 00:38:13.260090 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:13.260062    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf82be8f_4cfa_4545_aeaf_76d8ecb0d900.slice/crio-026ae18cd18a159bfe1c2ce2b9f37936a6979ed91b5c01272912c32cb6fe7011 WatchSource:0}: Error finding container 026ae18cd18a159bfe1c2ce2b9f37936a6979ed91b5c01272912c32cb6fe7011: Status 404 returned error can't find the container with id 026ae18cd18a159bfe1c2ce2b9f37936a6979ed91b5c01272912c32cb6fe7011
Aug 01 00:38:14.050345 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:14.050301    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t58fj" event={"ID":"8d0f282b-d478-4315-b903-74f767be07ae","Type":"ContainerStarted","Data":"098ddc78eeeb714e922a9c699a19042146b44b33709877332d1d89a19104a52e"}
Aug 01 00:38:14.053451 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:14.053401    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc" event={"ID":"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900","Type":"ContainerStarted","Data":"f929c16a73facd8b69acf7670f3b64c12745905881a3a0201ac338453911ddd1"}
Aug 01 00:38:14.053589 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:14.053508    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc" event={"ID":"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900","Type":"ContainerStarted","Data":"f0988b7763a25aa3e7ee02d8b8f7c41408c944806dfd3b1101d02b037a2b1f69"}
Aug 01 00:38:14.053589 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:14.053550    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc" event={"ID":"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900","Type":"ContainerStarted","Data":"026ae18cd18a159bfe1c2ce2b9f37936a6979ed91b5c01272912c32cb6fe7011"}
Aug 01 00:38:14.840721 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:14.840693    2570 patch_prober.go:28] interesting pod/image-registry-56c649478c-nzscm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Aug 01 00:38:14.841028 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:14.840745    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56c649478c-nzscm" podUID="1f014511-d183-4684-9e4d-3fb406908b03" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:38:15.057846 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:15.057811    2570 generic.go:356] "Generic (PLEG): container finished" podID="8d0f282b-d478-4315-b903-74f767be07ae" containerID="5728d4e7e1d3f27cde65fb0bc892922119a64683e0544ca300fc35546c6f949c" exitCode=0
Aug 01 00:38:15.058009 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:15.057902    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t58fj" event={"ID":"8d0f282b-d478-4315-b903-74f767be07ae","Type":"ContainerDied","Data":"5728d4e7e1d3f27cde65fb0bc892922119a64683e0544ca300fc35546c6f949c"}
Aug 01 00:38:15.059989 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:15.059965    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc" event={"ID":"bf82be8f-4cfa-4545-aeaf-76d8ecb0d900","Type":"ContainerStarted","Data":"216076ecd84b60208511aabdf08166ad698aa4ea146a9bab9b2e210ee0a063fe"}
Aug 01 00:38:15.101130 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:15.101077    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-7htgc" podStartSLOduration=1.974355569 podStartE2EDuration="3.101056794s" totalImagesPullingTime="1.126701225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:15.099579377 +0000 UTC m=+261.333511642" watchObservedRunningTime="2026-08-01 00:38:15.101056794 +0000 UTC m=+261.334989064"
Aug 01 00:38:16.064830 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.064795    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t58fj" event={"ID":"8d0f282b-d478-4315-b903-74f767be07ae","Type":"ContainerStarted","Data":"30a17eb8485b58470ad71a261bb91a2ab36af19d43a68eb73dfbce80b821493b"}
Aug 01 00:38:16.065320 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.064859    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t58fj" event={"ID":"8d0f282b-d478-4315-b903-74f767be07ae","Type":"ContainerStarted","Data":"8b40194ae1bf7c9cd1b8158e94c653a08657d69c6c60333fbd4f1e2aee92ce7a"}
Aug 01 00:38:16.084038 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.083996    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t58fj" podStartSLOduration=2.687096065 podStartE2EDuration="4.083979628s" totalImagesPullingTime="1.333679478s" totalInitContainerRuntime="63.204085ms" isStatefulPod=true podCreationTimestamp="2026-08-01 00:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:16.082271456 +0000 UTC m=+262.316203747" watchObservedRunningTime="2026-08-01 00:38:16.083979628 +0000 UTC m=+262.317911893"
Aug 01 00:38:16.532742 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.532667    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d789d7b79-t6xcx"]
Aug 01 00:38:16.535541 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.535522    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.538081 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.538058    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Aug 01 00:38:16.539153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.539129    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ffjp2\""
Aug 01 00:38:16.539267 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.539199    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 01 00:38:16.539267 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.539211    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Aug 01 00:38:16.539267 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.539199    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Aug 01 00:38:16.539412 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.539398    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2nlav82f7hich\""
Aug 01 00:38:16.548862 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.547271    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d789d7b79-t6xcx"]
Aug 01 00:38:16.652366 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.652338    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-client-ca-bundle\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.652501 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.652411    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d6de4a72-b91c-4031-9fd4-6dd92099ec54-audit-log\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.652501 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.652431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bqn\" (UniqueName: \"kubernetes.io/projected/d6de4a72-b91c-4031-9fd4-6dd92099ec54-kube-api-access-d8bqn\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.652501 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.652447    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-secret-metrics-server-tls\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.652501 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.652466    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6de4a72-b91c-4031-9fd4-6dd92099ec54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.652501 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.652485    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-secret-metrics-server-client-certs\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.652501 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.652503    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d6de4a72-b91c-4031-9fd4-6dd92099ec54-metrics-server-audit-profiles\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.753724 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.753698    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-secret-metrics-server-tls\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.753830 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.753728    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6de4a72-b91c-4031-9fd4-6dd92099ec54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.753830 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.753748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-secret-metrics-server-client-certs\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.753830 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.753766    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d6de4a72-b91c-4031-9fd4-6dd92099ec54-metrics-server-audit-profiles\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.753830 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.753783    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-client-ca-bundle\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.754033 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.753835    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d6de4a72-b91c-4031-9fd4-6dd92099ec54-audit-log\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.754033 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.753851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bqn\" (UniqueName: \"kubernetes.io/projected/d6de4a72-b91c-4031-9fd4-6dd92099ec54-kube-api-access-d8bqn\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.754410 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.754374    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d6de4a72-b91c-4031-9fd4-6dd92099ec54-audit-log\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.754669 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.754644    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6de4a72-b91c-4031-9fd4-6dd92099ec54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.754853 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.754837    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d6de4a72-b91c-4031-9fd4-6dd92099ec54-metrics-server-audit-profiles\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.756275 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.756253    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-secret-metrics-server-tls\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.756658 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.756641    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-client-ca-bundle\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.756824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.756795    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d6de4a72-b91c-4031-9fd4-6dd92099ec54-secret-metrics-server-client-certs\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.761433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.761407    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bqn\" (UniqueName: \"kubernetes.io/projected/d6de4a72-b91c-4031-9fd4-6dd92099ec54-kube-api-access-d8bqn\") pod \"metrics-server-6d789d7b79-t6xcx\" (UID: \"d6de4a72-b91c-4031-9fd4-6dd92099ec54\") " pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.845274 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.845252    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:16.986524 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:16.986490    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d789d7b79-t6xcx"]
Aug 01 00:38:16.989349 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:16.989230    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6de4a72_b91c_4031_9fd4_6dd92099ec54.slice/crio-b7d071691bfaa0b6edf7d399b96db8c57ad6cb720b3265ed4f9545660a66e317 WatchSource:0}: Error finding container b7d071691bfaa0b6edf7d399b96db8c57ad6cb720b3265ed4f9545660a66e317: Status 404 returned error can't find the container with id b7d071691bfaa0b6edf7d399b96db8c57ad6cb720b3265ed4f9545660a66e317
Aug 01 00:38:17.069398 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:17.069365    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx" event={"ID":"d6de4a72-b91c-4031-9fd4-6dd92099ec54","Type":"ContainerStarted","Data":"b7d071691bfaa0b6edf7d399b96db8c57ad6cb720b3265ed4f9545660a66e317"}
Aug 01 00:38:19.079598 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:19.079563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx" event={"ID":"d6de4a72-b91c-4031-9fd4-6dd92099ec54","Type":"ContainerStarted","Data":"29c524dab25af984cd1be8ff19132235c307c04b0eb9e6e9d5a5882ade9bb901"}
Aug 01 00:38:19.097238 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:19.097185    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx" podStartSLOduration=1.6419073709999998 podStartE2EDuration="3.097166194s" totalImagesPullingTime="1.455258823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:19.096637848 +0000 UTC m=+265.330570127" watchObservedRunningTime="2026-08-01 00:38:19.097166194 +0000 UTC m=+265.331098459"
Aug 01 00:38:24.841513 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:24.841482    2570 patch_prober.go:28] interesting pod/image-registry-56c649478c-nzscm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Aug 01 00:38:24.841861 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:24.841563    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56c649478c-nzscm" podUID="1f014511-d183-4684-9e4d-3fb406908b03" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:38:26.105778 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:26.105745    2570 generic.go:356] "Generic (PLEG): container finished" podID="cf1a71e4-e34e-4959-a913-91deeb31bda8" containerID="4ee949dab7798fd66322d2b4d7f0f48dfa18620d900fea8567ed0870838503cc" exitCode=255
Aug 01 00:38:26.106143 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:26.105817    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94" event={"ID":"cf1a71e4-e34e-4959-a913-91deeb31bda8","Type":"ContainerDied","Data":"4ee949dab7798fd66322d2b4d7f0f48dfa18620d900fea8567ed0870838503cc"}
Aug 01 00:38:26.111536 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:26.111518    2570 scope.go:117] "RemoveContainer" containerID="4ee949dab7798fd66322d2b4d7f0f48dfa18620d900fea8567ed0870838503cc"
Aug 01 00:38:27.110704 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:27.110669    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f6657556-q8n94" event={"ID":"cf1a71e4-e34e-4959-a913-91deeb31bda8","Type":"ContainerStarted","Data":"4415546c8544e2e8624e36e701d86669d282ae5c25d4ee599798495c0708151c"}
Aug 01 00:38:27.740606 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:27.740567    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg" podUID="a828bf62-04ca-40fa-928e-ea7080a7a1e7" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused"
Aug 01 00:38:28.115225 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:28.115194    2570 generic.go:356] "Generic (PLEG): container finished" podID="a828bf62-04ca-40fa-928e-ea7080a7a1e7" containerID="1ee577a69da65c1b5f71bb6cbb0f922a0c817001c25a3eb7a689e5faf291bcc9" exitCode=1
Aug 01 00:38:28.115591 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:28.115237    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg" event={"ID":"a828bf62-04ca-40fa-928e-ea7080a7a1e7","Type":"ContainerDied","Data":"1ee577a69da65c1b5f71bb6cbb0f922a0c817001c25a3eb7a689e5faf291bcc9"}
Aug 01 00:38:28.115591 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:28.115551    2570 scope.go:117] "RemoveContainer" containerID="1ee577a69da65c1b5f71bb6cbb0f922a0c817001c25a3eb7a689e5faf291bcc9"
Aug 01 00:38:29.120188 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:29.120148    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg" event={"ID":"a828bf62-04ca-40fa-928e-ea7080a7a1e7","Type":"ContainerStarted","Data":"091ff8ca96043252f71ae79eea0384865dcb9101097a5867c05015d934f687af"}
Aug 01 00:38:29.120558 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:29.120465    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:38:29.121009 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:29.120988    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654685d4b5-75pqg"
Aug 01 00:38:29.854434 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:29.854394    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56c649478c-nzscm" podUID="1f014511-d183-4684-9e4d-3fb406908b03" containerName="registry" containerID="cri-o://81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af" gracePeriod=30
Aug 01 00:38:30.095313 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.095288    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:38:30.125137 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.125064    2570 generic.go:356] "Generic (PLEG): container finished" podID="1f014511-d183-4684-9e4d-3fb406908b03" containerID="81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af" exitCode=0
Aug 01 00:38:30.125137 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.125107    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c649478c-nzscm"
Aug 01 00:38:30.125137 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.125120    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c649478c-nzscm" event={"ID":"1f014511-d183-4684-9e4d-3fb406908b03","Type":"ContainerDied","Data":"81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af"}
Aug 01 00:38:30.125597 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.125172    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c649478c-nzscm" event={"ID":"1f014511-d183-4684-9e4d-3fb406908b03","Type":"ContainerDied","Data":"3af8c12cc4d13e484af1c2ec5d0721c192cc66fa9367b8d185adc44571a0ee00"}
Aug 01 00:38:30.125597 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.125202    2570 scope.go:117] "RemoveContainer" containerID="81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af"
Aug 01 00:38:30.132971 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.132955    2570 scope.go:117] "RemoveContainer" containerID="81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af"
Aug 01 00:38:30.133263 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:38:30.133240    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af\": container with ID starting with 81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af not found: ID does not exist" containerID="81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af"
Aug 01 00:38:30.133314 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.133266    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af"} err="failed to get container status \"81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af\": rpc error: code = NotFound desc = could not find container \"81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af\": container with ID starting with 81d9ae58ea9c7793008359369d4a938fc4dd760358838ce5813419d5286a87af not found: ID does not exist"
Aug 01 00:38:30.250234 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250207    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-trusted-ca\") pod \"1f014511-d183-4684-9e4d-3fb406908b03\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") "
Aug 01 00:38:30.250386 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250249    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-image-registry-private-configuration\") pod \"1f014511-d183-4684-9e4d-3fb406908b03\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") "
Aug 01 00:38:30.250386 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250283    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvprn\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-kube-api-access-lvprn\") pod \"1f014511-d183-4684-9e4d-3fb406908b03\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") "
Aug 01 00:38:30.250386 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250318    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-installation-pull-secrets\") pod \"1f014511-d183-4684-9e4d-3fb406908b03\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") "
Aug 01 00:38:30.250386 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250345    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-registry-certificates\") pod \"1f014511-d183-4684-9e4d-3fb406908b03\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") "
Aug 01 00:38:30.250386 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250369    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") pod \"1f014511-d183-4684-9e4d-3fb406908b03\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") "
Aug 01 00:38:30.250638 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250472    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f014511-d183-4684-9e4d-3fb406908b03-ca-trust-extracted\") pod \"1f014511-d183-4684-9e4d-3fb406908b03\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") "
Aug 01 00:38:30.250638 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250518    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-bound-sa-token\") pod \"1f014511-d183-4684-9e4d-3fb406908b03\" (UID: \"1f014511-d183-4684-9e4d-3fb406908b03\") "
Aug 01 00:38:30.250741 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250635    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1f014511-d183-4684-9e4d-3fb406908b03" (UID: "1f014511-d183-4684-9e4d-3fb406908b03"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:38:30.250808 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.250787    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-trusted-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:38:30.251152 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.251093    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1f014511-d183-4684-9e4d-3fb406908b03" (UID: "1f014511-d183-4684-9e4d-3fb406908b03"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:38:30.252613 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.252586    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1f014511-d183-4684-9e4d-3fb406908b03" (UID: "1f014511-d183-4684-9e4d-3fb406908b03"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:38:30.252790 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.252764    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1f014511-d183-4684-9e4d-3fb406908b03" (UID: "1f014511-d183-4684-9e4d-3fb406908b03"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:38:30.252899 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.252885    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1f014511-d183-4684-9e4d-3fb406908b03" (UID: "1f014511-d183-4684-9e4d-3fb406908b03"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:38:30.252899 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.252887    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1f014511-d183-4684-9e4d-3fb406908b03" (UID: "1f014511-d183-4684-9e4d-3fb406908b03"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:38:30.253110 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.253092    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-kube-api-access-lvprn" (OuterVolumeSpecName: "kube-api-access-lvprn") pod "1f014511-d183-4684-9e4d-3fb406908b03" (UID: "1f014511-d183-4684-9e4d-3fb406908b03"). InnerVolumeSpecName "kube-api-access-lvprn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:38:30.259139 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.259113    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f014511-d183-4684-9e4d-3fb406908b03-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1f014511-d183-4684-9e4d-3fb406908b03" (UID: "1f014511-d183-4684-9e4d-3fb406908b03"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:38:30.351699 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.351675    2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f014511-d183-4684-9e4d-3fb406908b03-registry-certificates\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:38:30.351699 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.351696    2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-registry-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:38:30.351814 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.351705    2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f014511-d183-4684-9e4d-3fb406908b03-ca-trust-extracted\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:38:30.351814 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.351714    2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-bound-sa-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:38:30.351814 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.351723    2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-image-registry-private-configuration\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:38:30.351814 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.351731    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvprn\" (UniqueName: \"kubernetes.io/projected/1f014511-d183-4684-9e4d-3fb406908b03-kube-api-access-lvprn\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:38:30.351814 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.351741    2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f014511-d183-4684-9e4d-3fb406908b03-installation-pull-secrets\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:38:30.441339 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.441286    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56c649478c-nzscm"]
Aug 01 00:38:30.444690 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:30.444667    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56c649478c-nzscm"]
Aug 01 00:38:32.260730 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:32.260698    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f014511-d183-4684-9e4d-3fb406908b03" path="/var/lib/kubelet/pods/1f014511-d183-4684-9e4d-3fb406908b03/volumes"
Aug 01 00:38:34.750835 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:38:34.750796    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-r7rdc" podUID="d36e3319-9c8a-480e-aa41-4e7a98c0532a"
Aug 01 00:38:35.143437 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:35.143413    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:38:36.845987 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:36.845946    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:36.845987 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:36.846010    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:38.604914 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:38.604876    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:38:38.607301 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:38.607266    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d36e3319-9c8a-480e-aa41-4e7a98c0532a-metrics-tls\") pod \"dns-default-r7rdc\" (UID: \"d36e3319-9c8a-480e-aa41-4e7a98c0532a\") " pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:38:38.747377 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:38.747352    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qd5dz\""
Aug 01 00:38:38.755409 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:38.755393    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:38:38.883111 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:38.883084    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r7rdc"]
Aug 01 00:38:38.885329 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:38.885304    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36e3319_9c8a_480e_aa41_4e7a98c0532a.slice/crio-63550e721b5cf7a65ffd78703c18f4ee40928cf2d878c81c4f1531ac16d9f99c WatchSource:0}: Error finding container 63550e721b5cf7a65ffd78703c18f4ee40928cf2d878c81c4f1531ac16d9f99c: Status 404 returned error can't find the container with id 63550e721b5cf7a65ffd78703c18f4ee40928cf2d878c81c4f1531ac16d9f99c
Aug 01 00:38:39.156468 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:39.156389    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7rdc" event={"ID":"d36e3319-9c8a-480e-aa41-4e7a98c0532a","Type":"ContainerStarted","Data":"63550e721b5cf7a65ffd78703c18f4ee40928cf2d878c81c4f1531ac16d9f99c"}
Aug 01 00:38:41.165375 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:41.165342    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7rdc" event={"ID":"d36e3319-9c8a-480e-aa41-4e7a98c0532a","Type":"ContainerStarted","Data":"3ab81a55abcc3e546241a1be052cdad40842bb3e768e9972c153978355461772"}
Aug 01 00:38:41.165880 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:41.165404    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7rdc" event={"ID":"d36e3319-9c8a-480e-aa41-4e7a98c0532a","Type":"ContainerStarted","Data":"00de93bb306aa652832cc6a3238bf220b2217290ddd16be4bc66484f92563e0f"}
Aug 01 00:38:41.165880 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:41.165449    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:38:41.183310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:41.183260    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r7rdc" podStartSLOduration=251.567059432 podStartE2EDuration="4m13.183249729s" totalImagesPullingTime="1.616190297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:41.182923511 +0000 UTC m=+287.416855776" watchObservedRunningTime="2026-08-01 00:38:41.183249729 +0000 UTC m=+287.417181995"
Aug 01 00:38:44.762370 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.762332    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bb67b57f-xl5s6"]
Aug 01 00:38:44.762735 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.762595    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f014511-d183-4684-9e4d-3fb406908b03" containerName="registry"
Aug 01 00:38:44.762735 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.762606    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f014511-d183-4684-9e4d-3fb406908b03" containerName="registry"
Aug 01 00:38:44.762735 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.762653    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f014511-d183-4684-9e4d-3fb406908b03" containerName="registry"
Aug 01 00:38:44.767590 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.767573    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.771375 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.771350    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Aug 01 00:38:44.771513 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.771492    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Aug 01 00:38:44.771513 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.771504    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8tgrd\""
Aug 01 00:38:44.771663 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.771525    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 01 00:38:44.771663 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.771508    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Aug 01 00:38:44.771663 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.771428    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 01 00:38:44.771663 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.771434    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Aug 01 00:38:44.771961 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.771942    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Aug 01 00:38:44.775746 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.775727    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Aug 01 00:38:44.779202 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.779181    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb67b57f-xl5s6"]
Aug 01 00:38:44.851048 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.851028    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-serving-cert\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.851141 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.851052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-trusted-ca-bundle\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.851141 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.851078    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-oauth-config\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.851141 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.851099    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-service-ca\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.851244 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.851147    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-config\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.851244 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.851168    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-oauth-serving-cert\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.851244 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.851218    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkvzx\" (UniqueName: \"kubernetes.io/projected/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-kube-api-access-pkvzx\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.952196 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.952171    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkvzx\" (UniqueName: \"kubernetes.io/projected/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-kube-api-access-pkvzx\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.952302 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.952202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-serving-cert\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.952302 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.952217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-trusted-ca-bundle\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.952302 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.952241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-oauth-config\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.952302 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.952258    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-service-ca\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.952302 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.952284    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-config\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.952302 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.952304    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-oauth-serving-cert\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.952950 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.952906    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-service-ca\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.953082 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.953053    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-config\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.953182 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.953163    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-oauth-serving-cert\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.953253 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.953236    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-trusted-ca-bundle\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.954605 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.954581    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-serving-cert\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.954704 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.954685    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-oauth-config\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:44.960458 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:44.960439    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkvzx\" (UniqueName: \"kubernetes.io/projected/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-kube-api-access-pkvzx\") pod \"console-7bb67b57f-xl5s6\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") " pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:45.080067 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:45.080044    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:45.202661 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:45.202629    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb67b57f-xl5s6"]
Aug 01 00:38:45.206200 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:38:45.206173    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9626ec2_a4b4_4ca5_a8e7_4eec85b3ac4a.slice/crio-0664f4c00d8459487a87c56f0588dd8a11d71b56b565df1ec4d8811307758599 WatchSource:0}: Error finding container 0664f4c00d8459487a87c56f0588dd8a11d71b56b565df1ec4d8811307758599: Status 404 returned error can't find the container with id 0664f4c00d8459487a87c56f0588dd8a11d71b56b565df1ec4d8811307758599
Aug 01 00:38:46.184293 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:46.184248    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb67b57f-xl5s6" event={"ID":"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a","Type":"ContainerStarted","Data":"0664f4c00d8459487a87c56f0588dd8a11d71b56b565df1ec4d8811307758599"}
Aug 01 00:38:48.198946 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:48.198903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb67b57f-xl5s6" event={"ID":"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a","Type":"ContainerStarted","Data":"100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d"}
Aug 01 00:38:48.216474 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:48.216431    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bb67b57f-xl5s6" podStartSLOduration=1.313502542 podStartE2EDuration="4.216417078s" totalImagesPullingTime="2.902914536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:48.214996006 +0000 UTC m=+294.448928271" watchObservedRunningTime="2026-08-01 00:38:48.216417078 +0000 UTC m=+294.450349394"
Aug 01 00:38:51.171763 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:51.171732    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r7rdc"
Aug 01 00:38:54.154004 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:54.153977    2570 kubelet.go:1648] "Image garbage collection succeeded"
Aug 01 00:38:55.080240 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:55.080204    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:55.080240 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:55.080254    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:55.084973 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:55.084953    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:55.226798 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:55.226769    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:38:56.851717 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:56.851685    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:38:56.856723 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:38:56.856698    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d789d7b79-t6xcx"
Aug 01 00:39:15.780216 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:15.780177    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5fvlj_85d6afc8-b208-4fcf-b770-47f6560962a3/node-ca/0.log"
Aug 01 00:39:16.379210 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:16.379181    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5w4jn_a8d42714-2762-4696-8f20-c1005f745797/serve-healthcheck-canary/0.log"
Aug 01 00:39:39.582747 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.582709    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f696bcff6-v49n5"]
Aug 01 00:39:39.588249 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.588227    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.596653 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.596631    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f696bcff6-v49n5"]
Aug 01 00:39:39.735628 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.735586    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-trusted-ca-bundle\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.735628 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.735627    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-oauth-serving-cert\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.735865 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.735657    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-serving-cert\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.735865 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.735710    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-oauth-config\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.735865 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.735730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-service-ca\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.735865 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.735784    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjng\" (UniqueName: \"kubernetes.io/projected/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-kube-api-access-vsjng\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.735865 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.735825    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-config\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.836920 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.836828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-service-ca\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.836920 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.836884    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjng\" (UniqueName: \"kubernetes.io/projected/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-kube-api-access-vsjng\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.836920 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.836906    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-config\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.837109 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.836964    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-trusted-ca-bundle\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.837109 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.836978    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-oauth-serving-cert\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.837109 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.837091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-serving-cert\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.837225 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.837125    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-oauth-config\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.837748 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.837720    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-service-ca\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.837876 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.837729    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-oauth-serving-cert\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.837876 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.837766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-config\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.837876 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.837792    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-trusted-ca-bundle\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.839497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.839478    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-serving-cert\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.839632 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.839612    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-oauth-config\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.844305 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.844284    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjng\" (UniqueName: \"kubernetes.io/projected/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-kube-api-access-vsjng\") pod \"console-6f696bcff6-v49n5\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") " pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:39.897286 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:39.897258    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:40.018951 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:40.018905    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f696bcff6-v49n5"]
Aug 01 00:39:40.021528 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:39:40.021503    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a3ca68_dbc3_452d_b518_2bc70c0f3b64.slice/crio-f8a3c28d904800d974ecf2c80364b7b4f2f3014709c76a2345855237a2162d18 WatchSource:0}: Error finding container f8a3c28d904800d974ecf2c80364b7b4f2f3014709c76a2345855237a2162d18: Status 404 returned error can't find the container with id f8a3c28d904800d974ecf2c80364b7b4f2f3014709c76a2345855237a2162d18
Aug 01 00:39:40.384630 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:40.384595    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f696bcff6-v49n5" event={"ID":"30a3ca68-dbc3-452d-b518-2bc70c0f3b64","Type":"ContainerStarted","Data":"7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258"}
Aug 01 00:39:40.384801 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:40.384646    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f696bcff6-v49n5" event={"ID":"30a3ca68-dbc3-452d-b518-2bc70c0f3b64","Type":"ContainerStarted","Data":"f8a3c28d904800d974ecf2c80364b7b4f2f3014709c76a2345855237a2162d18"}
Aug 01 00:39:40.402120 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:40.402076    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f696bcff6-v49n5" podStartSLOduration=1.402064033 podStartE2EDuration="1.402064033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:40.400738919 +0000 UTC m=+346.634671182" watchObservedRunningTime="2026-08-01 00:39:40.402064033 +0000 UTC m=+346.635996298"
Aug 01 00:39:49.897902 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:49.897825    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:49.897902 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:49.897881    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:49.902595 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:49.902572    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:50.425341 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:50.425312    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:39:50.468467 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:39:50.468438    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb67b57f-xl5s6"]
Aug 01 00:40:15.490828 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.490767    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bb67b57f-xl5s6" podUID="c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" containerName="console" containerID="cri-o://100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d" gracePeriod=15
Aug 01 00:40:15.724062 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.724036    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:40:15.794400 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794329    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-serving-cert\") pod \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") "
Aug 01 00:40:15.794400 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794365    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-service-ca\") pod \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") "
Aug 01 00:40:15.794400 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794382    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-oauth-serving-cert\") pod \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") "
Aug 01 00:40:15.794671 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794408    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-trusted-ca-bundle\") pod \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") "
Aug 01 00:40:15.794671 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794584    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-oauth-config\") pod \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") "
Aug 01 00:40:15.794671 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794629    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-config\") pod \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") "
Aug 01 00:40:15.794826 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794685    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkvzx\" (UniqueName: \"kubernetes.io/projected/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-kube-api-access-pkvzx\") pod \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\" (UID: \"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a\") "
Aug 01 00:40:15.794826 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794786    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" (UID: "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:40:15.794926 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794818    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-service-ca" (OuterVolumeSpecName: "service-ca") pod "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" (UID: "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:40:15.794926 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794891    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:40:15.794926 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794908    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-oauth-serving-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:40:15.795118 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.794960    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" (UID: "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:40:15.795118 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.795073    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-config" (OuterVolumeSpecName: "console-config") pod "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" (UID: "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:40:15.796783 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.796750    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" (UID: "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:40:15.796882 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.796782    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" (UID: "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:40:15.796882 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.796831    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-kube-api-access-pkvzx" (OuterVolumeSpecName: "kube-api-access-pkvzx") pod "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" (UID: "c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a"). InnerVolumeSpecName "kube-api-access-pkvzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:40:15.895568 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.895544    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkvzx\" (UniqueName: \"kubernetes.io/projected/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-kube-api-access-pkvzx\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:40:15.895568 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.895568    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-serving-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:40:15.895734 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.895581    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-trusted-ca-bundle\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:40:15.895734 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.895598    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-oauth-config\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:40:15.895734 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:15.895609    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a-console-config\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:40:16.515980 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.515919    2570 generic.go:356] "Generic (PLEG): container finished" podID="c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" containerID="100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d" exitCode=0
Aug 01 00:40:16.516454 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.515990    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb67b57f-xl5s6"
Aug 01 00:40:16.516454 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.515983    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb67b57f-xl5s6" event={"ID":"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a","Type":"ContainerDied","Data":"100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d"}
Aug 01 00:40:16.516454 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.516130    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb67b57f-xl5s6" event={"ID":"c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a","Type":"ContainerDied","Data":"0664f4c00d8459487a87c56f0588dd8a11d71b56b565df1ec4d8811307758599"}
Aug 01 00:40:16.516454 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.516151    2570 scope.go:117] "RemoveContainer" containerID="100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d"
Aug 01 00:40:16.524736 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.524722    2570 scope.go:117] "RemoveContainer" containerID="100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d"
Aug 01 00:40:16.524995 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:40:16.524975    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d\": container with ID starting with 100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d not found: ID does not exist" containerID="100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d"
Aug 01 00:40:16.525039 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.525004    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d"} err="failed to get container status \"100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d\": rpc error: code = NotFound desc = could not find container \"100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d\": container with ID starting with 100ee34f7fb2710ae01850a18933b2ce1bca157d44a5e7aeb0d2aab940201e4d not found: ID does not exist"
Aug 01 00:40:16.532310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.532290    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb67b57f-xl5s6"]
Aug 01 00:40:16.534303 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:16.534283    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bb67b57f-xl5s6"]
Aug 01 00:40:18.260669 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:40:18.260634    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" path="/var/lib/kubelet/pods/c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a/volumes"
Aug 01 00:41:02.089643 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.089600    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c46f9d89d-85ncb"]
Aug 01 00:41:02.090223 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.089856    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" containerName="console"
Aug 01 00:41:02.090223 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.089867    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" containerName="console"
Aug 01 00:41:02.090223 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.089922    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9626ec2-a4b4-4ca5-a8e7-4eec85b3ac4a" containerName="console"
Aug 01 00:41:02.093788 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.093770    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.152388 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.152350    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c46f9d89d-85ncb"]
Aug 01 00:41:02.230769 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.230732    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29j27\" (UniqueName: \"kubernetes.io/projected/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-kube-api-access-29j27\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.230769 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.230774    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-oauth-serving-cert\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.231025 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.230800    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-service-ca\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.231025 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.230857    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-serving-cert\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.231025 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.230895    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-trusted-ca-bundle\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.231025 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.230919    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-oauth-config\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.231025 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.230953    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-config\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.331556 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.331516    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29j27\" (UniqueName: \"kubernetes.io/projected/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-kube-api-access-29j27\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.331556 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.331561    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-oauth-serving-cert\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.331824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.331583    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-service-ca\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.331824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.331600    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-serving-cert\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.331824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.331622    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-trusted-ca-bundle\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.331824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.331640    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-oauth-config\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.331824 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.331668    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-config\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.332452 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.332426    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-config\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.332562 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.332446    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-oauth-serving-cert\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.332562 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.332426    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-service-ca\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.332562 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.332551    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-trusted-ca-bundle\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.334264 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.334234    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-oauth-config\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.334357 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.334272    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-serving-cert\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.341355 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.341305    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29j27\" (UniqueName: \"kubernetes.io/projected/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-kube-api-access-29j27\") pod \"console-6c46f9d89d-85ncb\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") " pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.402362 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.402323    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:02.533020 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.532774    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c46f9d89d-85ncb"]
Aug 01 00:41:02.535495 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:41:02.535463    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226b0251_4ea1_4841_b4f3_31b4b5c9f2b7.slice/crio-be73f9ecb6e4214341fdd90b43fb55e227b907692f37263e4542cbb3f99c97d6 WatchSource:0}: Error finding container be73f9ecb6e4214341fdd90b43fb55e227b907692f37263e4542cbb3f99c97d6: Status 404 returned error can't find the container with id be73f9ecb6e4214341fdd90b43fb55e227b907692f37263e4542cbb3f99c97d6
Aug 01 00:41:02.675761 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.675680    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c46f9d89d-85ncb" event={"ID":"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7","Type":"ContainerStarted","Data":"bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a"}
Aug 01 00:41:02.675761 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.675737    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c46f9d89d-85ncb" event={"ID":"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7","Type":"ContainerStarted","Data":"be73f9ecb6e4214341fdd90b43fb55e227b907692f37263e4542cbb3f99c97d6"}
Aug 01 00:41:02.693690 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:02.693638    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c46f9d89d-85ncb" podStartSLOduration=0.69362312 podStartE2EDuration="693.62312ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:41:02.692901725 +0000 UTC m=+428.926833991" watchObservedRunningTime="2026-08-01 00:41:02.69362312 +0000 UTC m=+428.927555385"
Aug 01 00:41:12.403331 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:12.403296    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:12.403950 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:12.403360    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:12.408283 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:12.408261    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:12.714852 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:12.714770    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:41:12.758996 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:12.758969    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f696bcff6-v49n5"]
Aug 01 00:41:37.777725 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:37.777654    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f696bcff6-v49n5" podUID="30a3ca68-dbc3-452d-b518-2bc70c0f3b64" containerName="console" containerID="cri-o://7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258" gracePeriod=15
Aug 01 00:41:38.017314 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.017285    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:41:38.086374 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086343    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsjng\" (UniqueName: \"kubernetes.io/projected/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-kube-api-access-vsjng\") pod \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") "
Aug 01 00:41:38.086521 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086378    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-serving-cert\") pod \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") "
Aug 01 00:41:38.086521 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086395    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-service-ca\") pod \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") "
Aug 01 00:41:38.086521 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086413    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-oauth-serving-cert\") pod \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") "
Aug 01 00:41:38.086521 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086454    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-oauth-config\") pod \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") "
Aug 01 00:41:38.086521 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086499    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-config\") pod \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") "
Aug 01 00:41:38.086782 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086527    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-trusted-ca-bundle\") pod \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\" (UID: \"30a3ca68-dbc3-452d-b518-2bc70c0f3b64\") "
Aug 01 00:41:38.086961 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086908    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30a3ca68-dbc3-452d-b518-2bc70c0f3b64" (UID: "30a3ca68-dbc3-452d-b518-2bc70c0f3b64"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:41:38.086961 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.086920    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-service-ca" (OuterVolumeSpecName: "service-ca") pod "30a3ca68-dbc3-452d-b518-2bc70c0f3b64" (UID: "30a3ca68-dbc3-452d-b518-2bc70c0f3b64"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:41:38.087082 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.087001    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-config" (OuterVolumeSpecName: "console-config") pod "30a3ca68-dbc3-452d-b518-2bc70c0f3b64" (UID: "30a3ca68-dbc3-452d-b518-2bc70c0f3b64"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:41:38.087082 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.087018    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "30a3ca68-dbc3-452d-b518-2bc70c0f3b64" (UID: "30a3ca68-dbc3-452d-b518-2bc70c0f3b64"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:41:38.088558 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.088540    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-kube-api-access-vsjng" (OuterVolumeSpecName: "kube-api-access-vsjng") pod "30a3ca68-dbc3-452d-b518-2bc70c0f3b64" (UID: "30a3ca68-dbc3-452d-b518-2bc70c0f3b64"). InnerVolumeSpecName "kube-api-access-vsjng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:41:38.089019 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.089000    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30a3ca68-dbc3-452d-b518-2bc70c0f3b64" (UID: "30a3ca68-dbc3-452d-b518-2bc70c0f3b64"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:41:38.089066 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.089041    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30a3ca68-dbc3-452d-b518-2bc70c0f3b64" (UID: "30a3ca68-dbc3-452d-b518-2bc70c0f3b64"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:41:38.187518 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.187466    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-oauth-config\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:41:38.187518 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.187514    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-config\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:41:38.187518 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.187530    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-trusted-ca-bundle\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:41:38.187763 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.187542    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsjng\" (UniqueName: \"kubernetes.io/projected/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-kube-api-access-vsjng\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:41:38.187763 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.187555    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-console-serving-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:41:38.187763 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.187566    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:41:38.187763 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.187577    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30a3ca68-dbc3-452d-b518-2bc70c0f3b64-oauth-serving-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:41:38.803467 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.803427    2570 generic.go:356] "Generic (PLEG): container finished" podID="30a3ca68-dbc3-452d-b518-2bc70c0f3b64" containerID="7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258" exitCode=0
Aug 01 00:41:38.804153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.803474    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f696bcff6-v49n5"
Aug 01 00:41:38.804153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.803491    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f696bcff6-v49n5" event={"ID":"30a3ca68-dbc3-452d-b518-2bc70c0f3b64","Type":"ContainerDied","Data":"7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258"}
Aug 01 00:41:38.804153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.803540    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f696bcff6-v49n5" event={"ID":"30a3ca68-dbc3-452d-b518-2bc70c0f3b64","Type":"ContainerDied","Data":"f8a3c28d904800d974ecf2c80364b7b4f2f3014709c76a2345855237a2162d18"}
Aug 01 00:41:38.804153 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.803567    2570 scope.go:117] "RemoveContainer" containerID="7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258"
Aug 01 00:41:38.811617 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.811595    2570 scope.go:117] "RemoveContainer" containerID="7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258"
Aug 01 00:41:38.812031 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:41:38.812007    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258\": container with ID starting with 7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258 not found: ID does not exist" containerID="7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258"
Aug 01 00:41:38.812137 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.812037    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258"} err="failed to get container status \"7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258\": rpc error: code = NotFound desc = could not find container \"7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258\": container with ID starting with 7951feb91aaba0cd0632f6603656d5eca3a1dede60ef19a3c88986349b955258 not found: ID does not exist"
Aug 01 00:41:38.820834 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.820810    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f696bcff6-v49n5"]
Aug 01 00:41:38.825273 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:38.825250    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f696bcff6-v49n5"]
Aug 01 00:41:40.260256 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:41:40.260222    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a3ca68-dbc3-452d-b518-2bc70c0f3b64" path="/var/lib/kubelet/pods/30a3ca68-dbc3-452d-b518-2bc70c0f3b64/volumes"
Aug 01 00:42:04.697224 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.697193    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"]
Aug 01 00:42:04.697661 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.697432    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a3ca68-dbc3-452d-b518-2bc70c0f3b64" containerName="console"
Aug 01 00:42:04.697661 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.697442    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a3ca68-dbc3-452d-b518-2bc70c0f3b64" containerName="console"
Aug 01 00:42:04.697661 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.697499    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a3ca68-dbc3-452d-b518-2bc70c0f3b64" containerName="console"
Aug 01 00:42:04.703228 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.703198    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"
Aug 01 00:42:04.705844 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.705820    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Aug 01 00:42:04.705844 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.705834    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Aug 01 00:42:04.706036 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.705868    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lrg9j\""
Aug 01 00:42:04.717118 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.717099    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"]
Aug 01 00:42:04.767586 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.767565    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbadc9f4-d4d3-4779-89a2-b35f1470d023-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n74v5\" (UID: \"fbadc9f4-d4d3-4779-89a2-b35f1470d023\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"
Aug 01 00:42:04.767698 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.767612    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nt9\" (UniqueName: \"kubernetes.io/projected/fbadc9f4-d4d3-4779-89a2-b35f1470d023-kube-api-access-m6nt9\") pod \"cert-manager-operator-controller-manager-55c444bf86-n74v5\" (UID: \"fbadc9f4-d4d3-4779-89a2-b35f1470d023\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"
Aug 01 00:42:04.868777 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.868747    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nt9\" (UniqueName: \"kubernetes.io/projected/fbadc9f4-d4d3-4779-89a2-b35f1470d023-kube-api-access-m6nt9\") pod \"cert-manager-operator-controller-manager-55c444bf86-n74v5\" (UID: \"fbadc9f4-d4d3-4779-89a2-b35f1470d023\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"
Aug 01 00:42:04.868848 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.868799    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbadc9f4-d4d3-4779-89a2-b35f1470d023-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n74v5\" (UID: \"fbadc9f4-d4d3-4779-89a2-b35f1470d023\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"
Aug 01 00:42:04.869119 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.869105    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbadc9f4-d4d3-4779-89a2-b35f1470d023-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-n74v5\" (UID: \"fbadc9f4-d4d3-4779-89a2-b35f1470d023\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"
Aug 01 00:42:04.881410 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:04.881392    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nt9\" (UniqueName: \"kubernetes.io/projected/fbadc9f4-d4d3-4779-89a2-b35f1470d023-kube-api-access-m6nt9\") pod \"cert-manager-operator-controller-manager-55c444bf86-n74v5\" (UID: \"fbadc9f4-d4d3-4779-89a2-b35f1470d023\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"
Aug 01 00:42:05.012323 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:05.012264    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"
Aug 01 00:42:05.142253 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:05.142232    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5"]
Aug 01 00:42:05.145039 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:42:05.145007    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbadc9f4_d4d3_4779_89a2_b35f1470d023.slice/crio-3c609f811b5e5e4070878275c6c593cfd601c4ae7e2819e26c2808a299e5aad7 WatchSource:0}: Error finding container 3c609f811b5e5e4070878275c6c593cfd601c4ae7e2819e26c2808a299e5aad7: Status 404 returned error can't find the container with id 3c609f811b5e5e4070878275c6c593cfd601c4ae7e2819e26c2808a299e5aad7
Aug 01 00:42:05.147620 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:05.147600    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:42:05.898229 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:05.898193    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5" event={"ID":"fbadc9f4-d4d3-4779-89a2-b35f1470d023","Type":"ContainerStarted","Data":"3c609f811b5e5e4070878275c6c593cfd601c4ae7e2819e26c2808a299e5aad7"}
Aug 01 00:42:15.939702 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:15.939664    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5" event={"ID":"fbadc9f4-d4d3-4779-89a2-b35f1470d023","Type":"ContainerStarted","Data":"b9d7431b7d3f3014ef1265ef75925fa6004282ba5a1dbac64d50d10dab88ba87"}
Aug 01 00:42:15.963619 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:15.963566    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-n74v5" podStartSLOduration=1.924849177 podStartE2EDuration="11.963551058s" totalImagesPullingTime="10.038701881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:15.961470532 +0000 UTC m=+502.195402824" watchObservedRunningTime="2026-08-01 00:42:15.963551058 +0000 UTC m=+502.197483324"
Aug 01 00:42:17.072440 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.072407    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t2zq4"]
Aug 01 00:42:17.075601 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.075584    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:17.078040 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.078009    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Aug 01 00:42:17.078161 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.078113    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Aug 01 00:42:17.079197 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.079179    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nvq5m\""
Aug 01 00:42:17.085114 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.085094    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t2zq4"]
Aug 01 00:42:17.155811 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.155780    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgpm8\" (UniqueName: \"kubernetes.io/projected/24f913ff-1584-4528-a73f-c20e7c018cad-kube-api-access-cgpm8\") pod \"cert-manager-webhook-6486ffd997-t2zq4\" (UID: \"24f913ff-1584-4528-a73f-c20e7c018cad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:17.155980 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.155832    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24f913ff-1584-4528-a73f-c20e7c018cad-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t2zq4\" (UID: \"24f913ff-1584-4528-a73f-c20e7c018cad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:17.256593 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.256562    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgpm8\" (UniqueName: \"kubernetes.io/projected/24f913ff-1584-4528-a73f-c20e7c018cad-kube-api-access-cgpm8\") pod \"cert-manager-webhook-6486ffd997-t2zq4\" (UID: \"24f913ff-1584-4528-a73f-c20e7c018cad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:17.256740 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.256617    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24f913ff-1584-4528-a73f-c20e7c018cad-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t2zq4\" (UID: \"24f913ff-1584-4528-a73f-c20e7c018cad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:17.264724 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.264693    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24f913ff-1584-4528-a73f-c20e7c018cad-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t2zq4\" (UID: \"24f913ff-1584-4528-a73f-c20e7c018cad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:17.264838 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.264757    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgpm8\" (UniqueName: \"kubernetes.io/projected/24f913ff-1584-4528-a73f-c20e7c018cad-kube-api-access-cgpm8\") pod \"cert-manager-webhook-6486ffd997-t2zq4\" (UID: \"24f913ff-1584-4528-a73f-c20e7c018cad\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:17.385028 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.384924    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:17.514579 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.514489    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t2zq4"]
Aug 01 00:42:17.516679 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:42:17.516643    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24f913ff_1584_4528_a73f_c20e7c018cad.slice/crio-ca26fb66820487578195624af6110953108becdf0ebeba956a497de919dc560c WatchSource:0}: Error finding container ca26fb66820487578195624af6110953108becdf0ebeba956a497de919dc560c: Status 404 returned error can't find the container with id ca26fb66820487578195624af6110953108becdf0ebeba956a497de919dc560c
Aug 01 00:42:17.948559 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:17.948520    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4" event={"ID":"24f913ff-1584-4528-a73f-c20e7c018cad","Type":"ContainerStarted","Data":"ca26fb66820487578195624af6110953108becdf0ebeba956a497de919dc560c"}
Aug 01 00:42:21.966641 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:21.966554    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4" event={"ID":"24f913ff-1584-4528-a73f-c20e7c018cad","Type":"ContainerStarted","Data":"19dd7d953e4a8eef2184ed09a394e71c57c69ad8f923a31712600b4b04baeffc"}
Aug 01 00:42:21.967042 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:21.966637    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:21.984279 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:21.984235    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4" podStartSLOduration=0.898867379 podStartE2EDuration="4.984222384s" totalImagesPullingTime="4.085355005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:21.982320283 +0000 UTC m=+508.216252549" watchObservedRunningTime="2026-08-01 00:42:21.984222384 +0000 UTC m=+508.218154650"
Aug 01 00:42:27.055808 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.055778    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rf9tp"]
Aug 01 00:42:27.058896 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.058880    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rf9tp"
Aug 01 00:42:27.061154 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.061130    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jw6g8\""
Aug 01 00:42:27.067161 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.067140    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rf9tp"]
Aug 01 00:42:27.132349 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.132308    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8gq\" (UniqueName: \"kubernetes.io/projected/b154661a-ba54-4304-b619-0a035f719f30-kube-api-access-rw8gq\") pod \"cert-manager-55b4d954d5-rf9tp\" (UID: \"b154661a-ba54-4304-b619-0a035f719f30\") " pod="cert-manager/cert-manager-55b4d954d5-rf9tp"
Aug 01 00:42:27.132470 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.132375    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b154661a-ba54-4304-b619-0a035f719f30-bound-sa-token\") pod \"cert-manager-55b4d954d5-rf9tp\" (UID: \"b154661a-ba54-4304-b619-0a035f719f30\") " pod="cert-manager/cert-manager-55b4d954d5-rf9tp"
Aug 01 00:42:27.233155 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.233129    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8gq\" (UniqueName: \"kubernetes.io/projected/b154661a-ba54-4304-b619-0a035f719f30-kube-api-access-rw8gq\") pod \"cert-manager-55b4d954d5-rf9tp\" (UID: \"b154661a-ba54-4304-b619-0a035f719f30\") " pod="cert-manager/cert-manager-55b4d954d5-rf9tp"
Aug 01 00:42:27.233271 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.233174    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b154661a-ba54-4304-b619-0a035f719f30-bound-sa-token\") pod \"cert-manager-55b4d954d5-rf9tp\" (UID: \"b154661a-ba54-4304-b619-0a035f719f30\") " pod="cert-manager/cert-manager-55b4d954d5-rf9tp"
Aug 01 00:42:27.241275 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.241247    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b154661a-ba54-4304-b619-0a035f719f30-bound-sa-token\") pod \"cert-manager-55b4d954d5-rf9tp\" (UID: \"b154661a-ba54-4304-b619-0a035f719f30\") " pod="cert-manager/cert-manager-55b4d954d5-rf9tp"
Aug 01 00:42:27.241366 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.241325    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8gq\" (UniqueName: \"kubernetes.io/projected/b154661a-ba54-4304-b619-0a035f719f30-kube-api-access-rw8gq\") pod \"cert-manager-55b4d954d5-rf9tp\" (UID: \"b154661a-ba54-4304-b619-0a035f719f30\") " pod="cert-manager/cert-manager-55b4d954d5-rf9tp"
Aug 01 00:42:27.368601 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.368579    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rf9tp"
Aug 01 00:42:27.492868 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.492774    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rf9tp"]
Aug 01 00:42:27.495465 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:42:27.495439    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb154661a_ba54_4304_b619_0a035f719f30.slice/crio-f38b527cd3218807abf0c1efe5cc885eb043c88f1e281275cb56235178b71f8f WatchSource:0}: Error finding container f38b527cd3218807abf0c1efe5cc885eb043c88f1e281275cb56235178b71f8f: Status 404 returned error can't find the container with id f38b527cd3218807abf0c1efe5cc885eb043c88f1e281275cb56235178b71f8f
Aug 01 00:42:27.971951 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.971905    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-t2zq4"
Aug 01 00:42:27.990594 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.990563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rf9tp" event={"ID":"b154661a-ba54-4304-b619-0a035f719f30","Type":"ContainerStarted","Data":"b798cef46b820d5a6c8e881cbd689b0153bb685333f1f187423fb487c5e4f383"}
Aug 01 00:42:27.990594 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:27.990598    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rf9tp" event={"ID":"b154661a-ba54-4304-b619-0a035f719f30","Type":"ContainerStarted","Data":"f38b527cd3218807abf0c1efe5cc885eb043c88f1e281275cb56235178b71f8f"}
Aug 01 00:42:28.009917 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:28.009785    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rf9tp" podStartSLOduration=1.00976797 podStartE2EDuration="1.00976797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:28.009069829 +0000 UTC m=+514.243002097" watchObservedRunningTime="2026-08-01 00:42:28.00976797 +0000 UTC m=+514.243700236"
Aug 01 00:42:53.639658 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.639572    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"]
Aug 01 00:42:53.645670 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.645650    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.648920 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.648898    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Aug 01 00:42:53.649962 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.649943    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tj7sc\""
Aug 01 00:42:53.650407 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.650391    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Aug 01 00:42:53.650489 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.650421    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Aug 01 00:42:53.650489 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.650482    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Aug 01 00:42:53.665491 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.665470    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"]
Aug 01 00:42:53.829807 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.829774    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-manifests\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.829982 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.829818    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.829982 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.829865    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-tmp\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.829982 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.829908    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.830122 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.829980    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8r85\" (UniqueName: \"kubernetes.io/projected/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-kube-api-access-p8r85\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.930548 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.930487    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-manifests\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.930548 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.930522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.930669 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.930550    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-tmp\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.930669 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.930581    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.930669 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.930610    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8r85\" (UniqueName: \"kubernetes.io/projected/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-kube-api-access-p8r85\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.930914 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.930896    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-manifests\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.932975 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.932947    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-tmp\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.933072 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.933057    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.933193 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.933178    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.939815 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.939797    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8r85\" (UniqueName: \"kubernetes.io/projected/5c2ef3b9-05f2-47bb-ac1b-6f64858c0874-kube-api-access-p8r85\") pod \"opendatahub-operator-controller-manager-64f88dcc48-cm5hv\" (UID: \"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:53.955681 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:53.955665    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:54.096719 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:54.096600    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"]
Aug 01 00:42:54.099569 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:42:54.099541    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2ef3b9_05f2_47bb_ac1b_6f64858c0874.slice/crio-0e86a16541c008ece033cfe3e3f5a98727dcbbdf316e43ab7bb88814a6857c6f WatchSource:0}: Error finding container 0e86a16541c008ece033cfe3e3f5a98727dcbbdf316e43ab7bb88814a6857c6f: Status 404 returned error can't find the container with id 0e86a16541c008ece033cfe3e3f5a98727dcbbdf316e43ab7bb88814a6857c6f
Aug 01 00:42:55.096739 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:55.096700    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv" event={"ID":"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874","Type":"ContainerStarted","Data":"0e86a16541c008ece033cfe3e3f5a98727dcbbdf316e43ab7bb88814a6857c6f"}
Aug 01 00:42:57.106671 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:57.106635    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv" event={"ID":"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874","Type":"ContainerStarted","Data":"fb1e9a07df6dcbc10fb564bf3a42a8047fd92a11191480d1e90f3425a1a274d5"}
Aug 01 00:42:58.112108 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:58.112078    2570 generic.go:356] "Generic (PLEG): container finished" podID="5c2ef3b9-05f2-47bb-ac1b-6f64858c0874" containerID="fb1e9a07df6dcbc10fb564bf3a42a8047fd92a11191480d1e90f3425a1a274d5" exitCode=0
Aug 01 00:42:58.112501 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:58.112133    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv" event={"ID":"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874","Type":"ContainerDied","Data":"fb1e9a07df6dcbc10fb564bf3a42a8047fd92a11191480d1e90f3425a1a274d5"}
Aug 01 00:42:59.118322 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:59.118286    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv" event={"ID":"5c2ef3b9-05f2-47bb-ac1b-6f64858c0874","Type":"ContainerStarted","Data":"94a91533b92150eb73df4ba4734bba4405865ece548fd3052e4a4b4b668d631f"}
Aug 01 00:42:59.118790 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:59.118402    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:42:59.142560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:42:59.142504    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv" podStartSLOduration=1.806960136 podStartE2EDuration="6.142486362s" totalImagesPullingTime="3.275293209s" totalInitContainerRuntime="1.060233017s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:59.14088039 +0000 UTC m=+545.374812657" watchObservedRunningTime="2026-08-01 00:42:59.142486362 +0000 UTC m=+545.376418629"
Aug 01 00:43:10.124840 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:10.124807    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-cm5hv"
Aug 01 00:43:14.244606 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.244569    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2npxt"]
Aug 01 00:43:14.252746 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.252720    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:14.255349 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.255296    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Aug 01 00:43:14.255475 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.255404    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hc2hz\""
Aug 01 00:43:14.264433 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.264407    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2npxt"]
Aug 01 00:43:14.281804 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.281761    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8797bf-066c-4c26-9ffd-5d1382eee2b5-tls-certs\") pod \"model-serving-api-54d89bd5fc-2npxt\" (UID: \"7f8797bf-066c-4c26-9ffd-5d1382eee2b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:14.281986 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.281959    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmfw\" (UniqueName: \"kubernetes.io/projected/7f8797bf-066c-4c26-9ffd-5d1382eee2b5-kube-api-access-zwmfw\") pod \"model-serving-api-54d89bd5fc-2npxt\" (UID: \"7f8797bf-066c-4c26-9ffd-5d1382eee2b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:14.383341 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.383296    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8797bf-066c-4c26-9ffd-5d1382eee2b5-tls-certs\") pod \"model-serving-api-54d89bd5fc-2npxt\" (UID: \"7f8797bf-066c-4c26-9ffd-5d1382eee2b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:14.383544 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.383362    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmfw\" (UniqueName: \"kubernetes.io/projected/7f8797bf-066c-4c26-9ffd-5d1382eee2b5-kube-api-access-zwmfw\") pod \"model-serving-api-54d89bd5fc-2npxt\" (UID: \"7f8797bf-066c-4c26-9ffd-5d1382eee2b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:14.385796 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.385772    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8797bf-066c-4c26-9ffd-5d1382eee2b5-tls-certs\") pod \"model-serving-api-54d89bd5fc-2npxt\" (UID: \"7f8797bf-066c-4c26-9ffd-5d1382eee2b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:14.391254 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.391230    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmfw\" (UniqueName: \"kubernetes.io/projected/7f8797bf-066c-4c26-9ffd-5d1382eee2b5-kube-api-access-zwmfw\") pod \"model-serving-api-54d89bd5fc-2npxt\" (UID: \"7f8797bf-066c-4c26-9ffd-5d1382eee2b5\") " pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:14.566125 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.566094    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:14.697298 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:14.697262    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2npxt"]
Aug 01 00:43:14.700345 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:43:14.700317    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8797bf_066c_4c26_9ffd_5d1382eee2b5.slice/crio-77a217b5d00d16c692099db5070a76786e3922b6259b2c682dfdd2809457e78b WatchSource:0}: Error finding container 77a217b5d00d16c692099db5070a76786e3922b6259b2c682dfdd2809457e78b: Status 404 returned error can't find the container with id 77a217b5d00d16c692099db5070a76786e3922b6259b2c682dfdd2809457e78b
Aug 01 00:43:15.185684 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:15.185653    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2npxt" event={"ID":"7f8797bf-066c-4c26-9ffd-5d1382eee2b5","Type":"ContainerStarted","Data":"77a217b5d00d16c692099db5070a76786e3922b6259b2c682dfdd2809457e78b"}
Aug 01 00:43:18.203070 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:18.203028    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2npxt" event={"ID":"7f8797bf-066c-4c26-9ffd-5d1382eee2b5","Type":"ContainerStarted","Data":"2c6f6d797c138e37ac3c4be98eb28f641ad9239820c0b7f13185663c13db4298"}
Aug 01 00:43:18.203464 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:18.203160    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:18.219909 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:18.219839    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2npxt" podStartSLOduration=1.563498347 podStartE2EDuration="4.219825202s" totalImagesPullingTime="2.656326855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:18.219075962 +0000 UTC m=+564.453008228" watchObservedRunningTime="2026-08-01 00:43:18.219825202 +0000 UTC m=+564.453757469"
Aug 01 00:43:18.995261 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:18.995222    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"]
Aug 01 00:43:18.997856 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:18.997834    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.000460 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.000439    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Aug 01 00:43:19.000565 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.000472    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vzwfw\""
Aug 01 00:43:19.000565 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.000463    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Aug 01 00:43:19.001560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.001536    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 01 00:43:19.001657 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.001623    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 01 00:43:19.009371 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.009343    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"]
Aug 01 00:43:19.022383 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.022352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxnb\" (UniqueName: \"kubernetes.io/projected/4212e104-5c9d-49b3-8fca-3e69f1413f3a-kube-api-access-xtxnb\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.022524 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.022440    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tmp\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.022524 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.022495    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tls-certs\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.123466 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.123429    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tls-certs\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.123640 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.123483    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxnb\" (UniqueName: \"kubernetes.io/projected/4212e104-5c9d-49b3-8fca-3e69f1413f3a-kube-api-access-xtxnb\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.123640 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.123536    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tmp\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.123640 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:43:19.123563    2570 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Aug 01 00:43:19.123640 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:43:19.123627    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tls-certs podName:4212e104-5c9d-49b3-8fca-3e69f1413f3a nodeName:}" failed. No retries permitted until 2026-08-01 00:43:19.623610517 +0000 UTC m=+565.857542764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tls-certs") pod "kube-auth-proxy-7798dfdbd7-l7cgg" (UID: "4212e104-5c9d-49b3-8fca-3e69f1413f3a") : secret "kube-auth-proxy-tls" not found
Aug 01 00:43:19.125676 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.125659    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tmp\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.133812 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.133784    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxnb\" (UniqueName: \"kubernetes.io/projected/4212e104-5c9d-49b3-8fca-3e69f1413f3a-kube-api-access-xtxnb\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.627833 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.627796    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tls-certs\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.630575 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.630544    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4212e104-5c9d-49b3-8fca-3e69f1413f3a-tls-certs\") pod \"kube-auth-proxy-7798dfdbd7-l7cgg\" (UID: \"4212e104-5c9d-49b3-8fca-3e69f1413f3a\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:19.907353 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:19.907277    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"
Aug 01 00:43:20.047373 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:20.047240    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg"]
Aug 01 00:43:20.049679 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:43:20.049653    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4212e104_5c9d_49b3_8fca_3e69f1413f3a.slice/crio-2797b33b396f46fac12b776383f520d33f038565389bb0892b440a98becb48eb WatchSource:0}: Error finding container 2797b33b396f46fac12b776383f520d33f038565389bb0892b440a98becb48eb: Status 404 returned error can't find the container with id 2797b33b396f46fac12b776383f520d33f038565389bb0892b440a98becb48eb
Aug 01 00:43:20.213029 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:20.212947    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg" event={"ID":"4212e104-5c9d-49b3-8fca-3e69f1413f3a","Type":"ContainerStarted","Data":"2797b33b396f46fac12b776383f520d33f038565389bb0892b440a98becb48eb"}
Aug 01 00:43:21.238329 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.238293    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"]
Aug 01 00:43:21.241586 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.241554    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.243987 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.243970    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Aug 01 00:43:21.245601 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.245575    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Aug 01 00:43:21.245703 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.245612    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Aug 01 00:43:21.245810 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.245789    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Aug 01 00:43:21.246135 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.246118    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-k7gp7\""
Aug 01 00:43:21.246487 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.246463    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Aug 01 00:43:21.252138 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.252094    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"]
Aug 01 00:43:21.343062 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.343030    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2d50c4ac-3b53-4c30-9638-579fef54b213-manager-config\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.343246 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.343097    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72x8s\" (UniqueName: \"kubernetes.io/projected/2d50c4ac-3b53-4c30-9638-579fef54b213-kube-api-access-72x8s\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.343246 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.343158    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d50c4ac-3b53-4c30-9638-579fef54b213-metrics-cert\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.343246 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.343215    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d50c4ac-3b53-4c30-9638-579fef54b213-cert\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.444256 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.444217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72x8s\" (UniqueName: \"kubernetes.io/projected/2d50c4ac-3b53-4c30-9638-579fef54b213-kube-api-access-72x8s\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.444256 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.444261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d50c4ac-3b53-4c30-9638-579fef54b213-metrics-cert\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.444469 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.444291    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d50c4ac-3b53-4c30-9638-579fef54b213-cert\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.444469 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.444336    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2d50c4ac-3b53-4c30-9638-579fef54b213-manager-config\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.445045 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.445024    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2d50c4ac-3b53-4c30-9638-579fef54b213-manager-config\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.446744 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.446720    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d50c4ac-3b53-4c30-9638-579fef54b213-cert\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.446839 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.446755    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d50c4ac-3b53-4c30-9638-579fef54b213-metrics-cert\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.461758 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.461709    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72x8s\" (UniqueName: \"kubernetes.io/projected/2d50c4ac-3b53-4c30-9638-579fef54b213-kube-api-access-72x8s\") pod \"lws-controller-manager-8588bf7fd6-hr66k\" (UID: \"2d50c4ac-3b53-4c30-9638-579fef54b213\") " pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:21.554884 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:21.554817    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:22.276057 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:22.276036    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"]
Aug 01 00:43:22.277924 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:43:22.277903    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d50c4ac_3b53_4c30_9638_579fef54b213.slice/crio-d5f436375815009b9c5584d1fae9e0b28daa16373152d3b84d4f07a72fa37774 WatchSource:0}: Error finding container d5f436375815009b9c5584d1fae9e0b28daa16373152d3b84d4f07a72fa37774: Status 404 returned error can't find the container with id d5f436375815009b9c5584d1fae9e0b28daa16373152d3b84d4f07a72fa37774
Aug 01 00:43:23.237142 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:23.237071    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k" event={"ID":"2d50c4ac-3b53-4c30-9638-579fef54b213","Type":"ContainerStarted","Data":"d5f436375815009b9c5584d1fae9e0b28daa16373152d3b84d4f07a72fa37774"}
Aug 01 00:43:23.239194 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:23.239134    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg" event={"ID":"4212e104-5c9d-49b3-8fca-3e69f1413f3a","Type":"ContainerStarted","Data":"72fc92cd43d2f412014d1476b470b3139d48e1265b7de9a2b6c7596049d65f7e"}
Aug 01 00:43:23.296231 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:23.296169    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-l7cgg" podStartSLOduration=3.148520361 podStartE2EDuration="5.296149121s" totalImagesPullingTime="2.14762876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:23.295325239 +0000 UTC m=+569.529257505" watchObservedRunningTime="2026-08-01 00:43:23.296149121 +0000 UTC m=+569.530081386"
Aug 01 00:43:25.249306 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:25.249266    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k" event={"ID":"2d50c4ac-3b53-4c30-9638-579fef54b213","Type":"ContainerStarted","Data":"7cac38126526fed32f69164e5b4fc5443e7dce643cec37d1f25da439e4445384"}
Aug 01 00:43:25.249750 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:25.249411    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:25.267672 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:25.267622    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k" podStartSLOduration=1.768750352 podStartE2EDuration="4.267607513s" totalImagesPullingTime="2.498857161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:25.26697809 +0000 UTC m=+571.500910356" watchObservedRunningTime="2026-08-01 00:43:25.267607513 +0000 UTC m=+571.501539779"
Aug 01 00:43:29.211399 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:29.211370    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2npxt"
Aug 01 00:43:36.261245 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:36.261209    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8588bf7fd6-hr66k"
Aug 01 00:43:57.733653 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:43:57.733614    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c46f9d89d-85ncb"]
Aug 01 00:44:22.757538 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:22.757445    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c46f9d89d-85ncb" podUID="226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" containerName="console" containerID="cri-o://bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a" gracePeriod=15
Aug 01 00:44:22.990525 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:22.990500    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:44:23.055209 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055129    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-trusted-ca-bundle\") pod \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") "
Aug 01 00:44:23.055209 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055168    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-oauth-config\") pod \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") "
Aug 01 00:44:23.055209 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055196    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29j27\" (UniqueName: \"kubernetes.io/projected/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-kube-api-access-29j27\") pod \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") "
Aug 01 00:44:23.055475 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055218    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-config\") pod \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") "
Aug 01 00:44:23.055475 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055237    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-oauth-serving-cert\") pod \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") "
Aug 01 00:44:23.055475 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055283    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-serving-cert\") pod \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") "
Aug 01 00:44:23.055475 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055313    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-service-ca\") pod \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\" (UID: \"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7\") "
Aug 01 00:44:23.055782 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055747    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-config" (OuterVolumeSpecName: "console-config") pod "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" (UID: "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:44:23.055863 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055785    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-service-ca" (OuterVolumeSpecName: "service-ca") pod "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" (UID: "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:44:23.055863 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055749    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" (UID: "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:44:23.055863 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.055758    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" (UID: "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:44:23.057331 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.057296    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" (UID: "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:44:23.057802 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.057779    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" (UID: "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:44:23.057802 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.057789    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-kube-api-access-29j27" (OuterVolumeSpecName: "kube-api-access-29j27") pod "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" (UID: "226b0251-4ea1-4841-b4f3-31b4b5c9f2b7"). InnerVolumeSpecName "kube-api-access-29j27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:44:23.156199 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.156163    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-serving-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:44:23.156199 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.156191    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:44:23.156199 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.156200    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-trusted-ca-bundle\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:44:23.156199 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.156209    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-oauth-config\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:44:23.156456 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.156218    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29j27\" (UniqueName: \"kubernetes.io/projected/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-kube-api-access-29j27\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:44:23.156456 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.156226    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-console-config\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:44:23.156456 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.156234    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7-oauth-serving-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:44:23.495254 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.495222    2570 generic.go:356] "Generic (PLEG): container finished" podID="226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" containerID="bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a" exitCode=0
Aug 01 00:44:23.495431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.495278    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c46f9d89d-85ncb"
Aug 01 00:44:23.495431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.495317    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c46f9d89d-85ncb" event={"ID":"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7","Type":"ContainerDied","Data":"bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a"}
Aug 01 00:44:23.495431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.495370    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c46f9d89d-85ncb" event={"ID":"226b0251-4ea1-4841-b4f3-31b4b5c9f2b7","Type":"ContainerDied","Data":"be73f9ecb6e4214341fdd90b43fb55e227b907692f37263e4542cbb3f99c97d6"}
Aug 01 00:44:23.495431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.495391    2570 scope.go:117] "RemoveContainer" containerID="bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a"
Aug 01 00:44:23.503845 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.503826    2570 scope.go:117] "RemoveContainer" containerID="bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a"
Aug 01 00:44:23.504171 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:44:23.504150    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a\": container with ID starting with bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a not found: ID does not exist" containerID="bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a"
Aug 01 00:44:23.504219 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.504181    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a"} err="failed to get container status \"bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a\": rpc error: code = NotFound desc = could not find container \"bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a\": container with ID starting with bb981211bd01786c32e65f42f3920ddda9b4f48ba81d00eb82f9bb893bfbb26a not found: ID does not exist"
Aug 01 00:44:23.524075 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.524051    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c46f9d89d-85ncb"]
Aug 01 00:44:23.527810 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:23.527788    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c46f9d89d-85ncb"]
Aug 01 00:44:24.261145 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:24.261101    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" path="/var/lib/kubelet/pods/226b0251-4ea1-4841-b4f3-31b4b5c9f2b7/volumes"
Aug 01 00:44:57.153658 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.153619    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"]
Aug 01 00:44:57.154092 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.153948    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" containerName="console"
Aug 01 00:44:57.154092 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.153974    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" containerName="console"
Aug 01 00:44:57.154092 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.154031    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="226b0251-4ea1-4841-b4f3-31b4b5c9f2b7" containerName="console"
Aug 01 00:44:57.157747 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.157729    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:44:57.160360 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.160340    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 01 00:44:57.160482 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.160369    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 01 00:44:57.160482 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.160372    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8zn4g\""
Aug 01 00:44:57.171688 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.171648    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"]
Aug 01 00:44:57.236608 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.236574    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8x2b\" (UniqueName: \"kubernetes.io/projected/55014818-e553-4e35-b8b3-6ecb9f72f664-kube-api-access-k8x2b\") pod \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" (UID: \"55014818-e553-4e35-b8b3-6ecb9f72f664\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:44:57.236779 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.236629    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55014818-e553-4e35-b8b3-6ecb9f72f664-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" (UID: \"55014818-e553-4e35-b8b3-6ecb9f72f664\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:44:57.337354 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.337323    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55014818-e553-4e35-b8b3-6ecb9f72f664-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" (UID: \"55014818-e553-4e35-b8b3-6ecb9f72f664\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:44:57.337510 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.337384    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8x2b\" (UniqueName: \"kubernetes.io/projected/55014818-e553-4e35-b8b3-6ecb9f72f664-kube-api-access-k8x2b\") pod \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" (UID: \"55014818-e553-4e35-b8b3-6ecb9f72f664\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:44:57.337705 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.337684    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55014818-e553-4e35-b8b3-6ecb9f72f664-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" (UID: \"55014818-e553-4e35-b8b3-6ecb9f72f664\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:44:57.347503 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.347479    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8x2b\" (UniqueName: \"kubernetes.io/projected/55014818-e553-4e35-b8b3-6ecb9f72f664-kube-api-access-k8x2b\") pod \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" (UID: \"55014818-e553-4e35-b8b3-6ecb9f72f664\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:44:57.468247 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.468167    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:44:57.603909 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.603869    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"]
Aug 01 00:44:57.606952 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:44:57.606912    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55014818_e553_4e35_b8b3_6ecb9f72f664.slice/crio-58d7bc57795857cd8077b8a007f5f849abcbd937275f5baaa80c7d20d1b1f504 WatchSource:0}: Error finding container 58d7bc57795857cd8077b8a007f5f849abcbd937275f5baaa80c7d20d1b1f504: Status 404 returned error can't find the container with id 58d7bc57795857cd8077b8a007f5f849abcbd937275f5baaa80c7d20d1b1f504
Aug 01 00:44:57.635793 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.635766    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6" event={"ID":"55014818-e553-4e35-b8b3-6ecb9f72f664","Type":"ContainerStarted","Data":"58d7bc57795857cd8077b8a007f5f849abcbd937275f5baaa80c7d20d1b1f504"}
Aug 01 00:44:57.844823 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.844790    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"]
Aug 01 00:44:57.851332 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.851305    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"]
Aug 01 00:44:57.876817 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.876785    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"]
Aug 01 00:44:57.881336 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.881296    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:44:57.894857 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:57.894830    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"]
Aug 01 00:44:58.043119 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.043083    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzqm\" (UniqueName: \"kubernetes.io/projected/8c7aae4d-e412-407b-92ae-798510473232-kube-api-access-9wzqm\") pod \"kuadrant-operator-controller-manager-96db98c47-bl4qm\" (UID: \"8c7aae4d-e412-407b-92ae-798510473232\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:44:58.043291 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.043186    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c7aae4d-e412-407b-92ae-798510473232-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-bl4qm\" (UID: \"8c7aae4d-e412-407b-92ae-798510473232\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:44:58.144313 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.144223    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c7aae4d-e412-407b-92ae-798510473232-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-bl4qm\" (UID: \"8c7aae4d-e412-407b-92ae-798510473232\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:44:58.144313 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.144304    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzqm\" (UniqueName: \"kubernetes.io/projected/8c7aae4d-e412-407b-92ae-798510473232-kube-api-access-9wzqm\") pod \"kuadrant-operator-controller-manager-96db98c47-bl4qm\" (UID: \"8c7aae4d-e412-407b-92ae-798510473232\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:44:58.144637 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.144615    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c7aae4d-e412-407b-92ae-798510473232-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-bl4qm\" (UID: \"8c7aae4d-e412-407b-92ae-798510473232\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:44:58.156387 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.156359    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzqm\" (UniqueName: \"kubernetes.io/projected/8c7aae4d-e412-407b-92ae-798510473232-kube-api-access-9wzqm\") pod \"kuadrant-operator-controller-manager-96db98c47-bl4qm\" (UID: \"8c7aae4d-e412-407b-92ae-798510473232\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:44:58.192426 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.192401    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:44:58.323043 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.323022    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"]
Aug 01 00:44:58.325354 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:44:58.325329    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7aae4d_e412_407b_92ae_798510473232.slice/crio-bdd940edb8d4ba481fb9a7858deb0bd8bbbb024df28141bf42cd9ad4733a3ff2 WatchSource:0}: Error finding container bdd940edb8d4ba481fb9a7858deb0bd8bbbb024df28141bf42cd9ad4733a3ff2: Status 404 returned error can't find the container with id bdd940edb8d4ba481fb9a7858deb0bd8bbbb024df28141bf42cd9ad4733a3ff2
Aug 01 00:44:58.643306 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:44:58.643271    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm" event={"ID":"8c7aae4d-e412-407b-92ae-798510473232","Type":"ContainerStarted","Data":"bdd940edb8d4ba481fb9a7858deb0bd8bbbb024df28141bf42cd9ad4733a3ff2"}
Aug 01 00:45:02.668045 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:02.667920    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" containerName="manager" containerID="cri-o://e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57" gracePeriod=2
Aug 01 00:45:02.910347 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:02.910311    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:45:02.912823 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:45:02.912791    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" is forbidden: User \"system:node:ip-10-0-141-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-142.ec2.internal' and this object" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:45:03.090138 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.090105    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8x2b\" (UniqueName: \"kubernetes.io/projected/55014818-e553-4e35-b8b3-6ecb9f72f664-kube-api-access-k8x2b\") pod \"55014818-e553-4e35-b8b3-6ecb9f72f664\" (UID: \"55014818-e553-4e35-b8b3-6ecb9f72f664\") "
Aug 01 00:45:03.090324 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.090166    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55014818-e553-4e35-b8b3-6ecb9f72f664-extensions-socket-volume\") pod \"55014818-e553-4e35-b8b3-6ecb9f72f664\" (UID: \"55014818-e553-4e35-b8b3-6ecb9f72f664\") "
Aug 01 00:45:03.090438 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.090402    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55014818-e553-4e35-b8b3-6ecb9f72f664-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "55014818-e553-4e35-b8b3-6ecb9f72f664" (UID: "55014818-e553-4e35-b8b3-6ecb9f72f664"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:45:03.151624 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.151579    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55014818-e553-4e35-b8b3-6ecb9f72f664-kube-api-access-k8x2b" (OuterVolumeSpecName: "kube-api-access-k8x2b") pod "55014818-e553-4e35-b8b3-6ecb9f72f664" (UID: "55014818-e553-4e35-b8b3-6ecb9f72f664"). InnerVolumeSpecName "kube-api-access-k8x2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:45:03.191616 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.191584    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55014818-e553-4e35-b8b3-6ecb9f72f664-extensions-socket-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:45:03.191616 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.191613    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8x2b\" (UniqueName: \"kubernetes.io/projected/55014818-e553-4e35-b8b3-6ecb9f72f664-kube-api-access-k8x2b\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:45:03.674704 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.674617    2570 generic.go:356] "Generic (PLEG): container finished" podID="55014818-e553-4e35-b8b3-6ecb9f72f664" containerID="e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57" exitCode=2
Aug 01 00:45:03.674704 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.674657    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:45:03.675232 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.674723    2570 scope.go:117] "RemoveContainer" containerID="e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57"
Aug 01 00:45:03.676628 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.676604    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm" event={"ID":"8c7aae4d-e412-407b-92ae-798510473232","Type":"ContainerStarted","Data":"aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1"}
Aug 01 00:45:03.676743 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.676723    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:45:03.677043 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:45:03.677023    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" is forbidden: User \"system:node:ip-10-0-141-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-142.ec2.internal' and this object" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:45:03.678811 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:45:03.678792    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" is forbidden: User \"system:node:ip-10-0-141-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-142.ec2.internal' and this object" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:45:03.684614 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.684600    2570 scope.go:117] "RemoveContainer" containerID="e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57"
Aug 01 00:45:03.684867 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:45:03.684847    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57\": container with ID starting with e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57 not found: ID does not exist" containerID="e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57"
Aug 01 00:45:03.684924 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.684874    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57"} err="failed to get container status \"e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57\": rpc error: code = NotFound desc = could not find container \"e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57\": container with ID starting with e0956406388cbb7ee20ad228d29701e267f28055b26adb248bd5e309467f2f57 not found: ID does not exist"
Aug 01 00:45:03.700920 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:03.700880    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm" podStartSLOduration=2.262459181 podStartE2EDuration="6.700867721s" totalImagesPullingTime="4.43840854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:03.699386665 +0000 UTC m=+669.933318929" watchObservedRunningTime="2026-08-01 00:45:03.700867721 +0000 UTC m=+669.934799988"
Aug 01 00:45:03.702153 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:45:03.702131    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" is forbidden: User \"system:node:ip-10-0-141-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-142.ec2.internal' and this object" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:45:04.260661 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:45:04.260624    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-p4bj6\" is forbidden: User \"system:node:ip-10-0-141-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-142.ec2.internal' and this object" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-p4bj6"
Aug 01 00:45:04.262460 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:04.262430    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" path="/var/lib/kubelet/pods/55014818-e553-4e35-b8b3-6ecb9f72f664/volumes"
Aug 01 00:45:14.683731 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:14.683699    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:45:27.259264 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.259231    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"]
Aug 01 00:45:27.259782 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.259417    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm" podUID="8c7aae4d-e412-407b-92ae-798510473232" containerName="manager" containerID="cri-o://aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1" gracePeriod=10
Aug 01 00:45:27.498923 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.498899    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:45:27.585977 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.585953    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c7aae4d-e412-407b-92ae-798510473232-extensions-socket-volume\") pod \"8c7aae4d-e412-407b-92ae-798510473232\" (UID: \"8c7aae4d-e412-407b-92ae-798510473232\") "
Aug 01 00:45:27.586140 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.586017    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wzqm\" (UniqueName: \"kubernetes.io/projected/8c7aae4d-e412-407b-92ae-798510473232-kube-api-access-9wzqm\") pod \"8c7aae4d-e412-407b-92ae-798510473232\" (UID: \"8c7aae4d-e412-407b-92ae-798510473232\") "
Aug 01 00:45:27.586404 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.586380    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7aae4d-e412-407b-92ae-798510473232-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8c7aae4d-e412-407b-92ae-798510473232" (UID: "8c7aae4d-e412-407b-92ae-798510473232"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:45:27.587964 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.587926    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7aae4d-e412-407b-92ae-798510473232-kube-api-access-9wzqm" (OuterVolumeSpecName: "kube-api-access-9wzqm") pod "8c7aae4d-e412-407b-92ae-798510473232" (UID: "8c7aae4d-e412-407b-92ae-798510473232"). InnerVolumeSpecName "kube-api-access-9wzqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:45:27.687021 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.686997    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c7aae4d-e412-407b-92ae-798510473232-extensions-socket-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:45:27.687021 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.687018    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzqm\" (UniqueName: \"kubernetes.io/projected/8c7aae4d-e412-407b-92ae-798510473232-kube-api-access-9wzqm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:45:27.786004 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.785965    2570 generic.go:356] "Generic (PLEG): container finished" podID="8c7aae4d-e412-407b-92ae-798510473232" containerID="aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1" exitCode=0
Aug 01 00:45:27.786004 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.786004    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm" event={"ID":"8c7aae4d-e412-407b-92ae-798510473232","Type":"ContainerDied","Data":"aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1"}
Aug 01 00:45:27.786224 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.786075    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"
Aug 01 00:45:27.786224 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.786097    2570 scope.go:117] "RemoveContainer" containerID="aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1"
Aug 01 00:45:27.786224 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.786083    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm" event={"ID":"8c7aae4d-e412-407b-92ae-798510473232","Type":"ContainerDied","Data":"bdd940edb8d4ba481fb9a7858deb0bd8bbbb024df28141bf42cd9ad4733a3ff2"}
Aug 01 00:45:27.794505 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.794489    2570 scope.go:117] "RemoveContainer" containerID="aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1"
Aug 01 00:45:27.794725 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:45:27.794704    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1\": container with ID starting with aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1 not found: ID does not exist" containerID="aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1"
Aug 01 00:45:27.794766 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.794732    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1"} err="failed to get container status \"aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1\": rpc error: code = NotFound desc = could not find container \"aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1\": container with ID starting with aa553b7bc60b465a9533b81d76ec39aaaac5200a41a5672afcf1b5c4ca3376c1 not found: ID does not exist"
Aug 01 00:45:27.807025 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.806980    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"]
Aug 01 00:45:27.824990 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:27.824966    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-bl4qm"]
Aug 01 00:45:28.262133 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:28.262103    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7aae4d-e412-407b-92ae-798510473232" path="/var/lib/kubelet/pods/8c7aae4d-e412-407b-92ae-798510473232/volumes"
Aug 01 00:45:45.804711 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.804678    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"]
Aug 01 00:45:45.805170 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.805008    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7aae4d-e412-407b-92ae-798510473232" containerName="manager"
Aug 01 00:45:45.805170 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.805021    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7aae4d-e412-407b-92ae-798510473232" containerName="manager"
Aug 01 00:45:45.805170 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.805030    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" containerName="manager"
Aug 01 00:45:45.805170 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.805035    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" containerName="manager"
Aug 01 00:45:45.805170 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.805096    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7aae4d-e412-407b-92ae-798510473232" containerName="manager"
Aug 01 00:45:45.805170 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.805104    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="55014818-e553-4e35-b8b3-6ecb9f72f664" containerName="manager"
Aug 01 00:45:45.809560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.809542    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:45.812164 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.812142    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 01 00:45:45.812253 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.812144    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Aug 01 00:45:45.813178 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.813159    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-677vz\""
Aug 01 00:45:45.813316 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.813167    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 01 00:45:45.815242 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.815221    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"]
Aug 01 00:45:45.897525 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.897491    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"]
Aug 01 00:45:45.935213 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.935186    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-config-file\") pod \"limitador-limitador-859c4d96d6-wcmdp\" (UID: \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:45.935346 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:45.935236    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55725\" (UniqueName: \"kubernetes.io/projected/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-kube-api-access-55725\") pod \"limitador-limitador-859c4d96d6-wcmdp\" (UID: \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:46.035607 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.035576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55725\" (UniqueName: \"kubernetes.io/projected/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-kube-api-access-55725\") pod \"limitador-limitador-859c4d96d6-wcmdp\" (UID: \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:46.035794 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.035659    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-config-file\") pod \"limitador-limitador-859c4d96d6-wcmdp\" (UID: \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:46.036303 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.036278    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-config-file\") pod \"limitador-limitador-859c4d96d6-wcmdp\" (UID: \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:46.046787 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.046753    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55725\" (UniqueName: \"kubernetes.io/projected/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-kube-api-access-55725\") pod \"limitador-limitador-859c4d96d6-wcmdp\" (UID: \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:46.120712 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.120670    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:46.253777 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.253745    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"]
Aug 01 00:45:46.257354 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:45:46.257321    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3dfd9f4_7999_41b7_a3e0_90dc7beb0b41.slice/crio-4ea4274f1616767e54504c58ecb7fd83f5f86731e7dc3a14aeebe3b30862c3bc WatchSource:0}: Error finding container 4ea4274f1616767e54504c58ecb7fd83f5f86731e7dc3a14aeebe3b30862c3bc: Status 404 returned error can't find the container with id 4ea4274f1616767e54504c58ecb7fd83f5f86731e7dc3a14aeebe3b30862c3bc
Aug 01 00:45:46.559460 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.559387    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-tdr8r"]
Aug 01 00:45:46.562239 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.562219    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-tdr8r"
Aug 01 00:45:46.564611 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.564588    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5lsdz\""
Aug 01 00:45:46.570290 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.570268    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tdr8r"]
Aug 01 00:45:46.640902 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.640866    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2cx\" (UniqueName: \"kubernetes.io/projected/8a895e7a-fde0-4946-b109-7f313e2f686a-kube-api-access-gz2cx\") pod \"authorino-b54df94b4-tdr8r\" (UID: \"8a895e7a-fde0-4946-b109-7f313e2f686a\") " pod="kuadrant-system/authorino-b54df94b4-tdr8r"
Aug 01 00:45:46.742061 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.742019    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2cx\" (UniqueName: \"kubernetes.io/projected/8a895e7a-fde0-4946-b109-7f313e2f686a-kube-api-access-gz2cx\") pod \"authorino-b54df94b4-tdr8r\" (UID: \"8a895e7a-fde0-4946-b109-7f313e2f686a\") " pod="kuadrant-system/authorino-b54df94b4-tdr8r"
Aug 01 00:45:46.749768 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.749739    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2cx\" (UniqueName: \"kubernetes.io/projected/8a895e7a-fde0-4946-b109-7f313e2f686a-kube-api-access-gz2cx\") pod \"authorino-b54df94b4-tdr8r\" (UID: \"8a895e7a-fde0-4946-b109-7f313e2f686a\") " pod="kuadrant-system/authorino-b54df94b4-tdr8r"
Aug 01 00:45:46.866991 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.866954    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp" event={"ID":"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41","Type":"ContainerStarted","Data":"4ea4274f1616767e54504c58ecb7fd83f5f86731e7dc3a14aeebe3b30862c3bc"}
Aug 01 00:45:46.873133 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.873113    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-tdr8r"
Aug 01 00:45:46.994788 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:46.994719    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tdr8r"]
Aug 01 00:45:46.996797 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:45:46.996773    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a895e7a_fde0_4946_b109_7f313e2f686a.slice/crio-56ffdac19411ad9b65e464a5a6844b2949ab506f83e90bd3b5510d6a5af4a191 WatchSource:0}: Error finding container 56ffdac19411ad9b65e464a5a6844b2949ab506f83e90bd3b5510d6a5af4a191: Status 404 returned error can't find the container with id 56ffdac19411ad9b65e464a5a6844b2949ab506f83e90bd3b5510d6a5af4a191
Aug 01 00:45:47.876417 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:47.876373    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-tdr8r" event={"ID":"8a895e7a-fde0-4946-b109-7f313e2f686a","Type":"ContainerStarted","Data":"56ffdac19411ad9b65e464a5a6844b2949ab506f83e90bd3b5510d6a5af4a191"}
Aug 01 00:45:47.877992 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:47.877960    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp" event={"ID":"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41","Type":"ContainerStarted","Data":"9bbc7eb3f10543b61c41b289967eea6a9202c127e7ab1eb3c4386a8572a9f841"}
Aug 01 00:45:47.878116 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:47.878097    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:47.893737 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:47.893684    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp" podStartSLOduration=1.474084038 podStartE2EDuration="2.89366835s" totalImagesPullingTime="1.419584312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:47.892536031 +0000 UTC m=+714.126468298" watchObservedRunningTime="2026-08-01 00:45:47.89366835 +0000 UTC m=+714.127600616"
Aug 01 00:45:49.445258 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:49.445217    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tdr8r"]
Aug 01 00:45:50.896209 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:50.896168    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-tdr8r" event={"ID":"8a895e7a-fde0-4946-b109-7f313e2f686a","Type":"ContainerStarted","Data":"10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359"}
Aug 01 00:45:50.896640 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:50.896227    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-tdr8r" podUID="8a895e7a-fde0-4946-b109-7f313e2f686a" containerName="authorino" containerID="cri-o://10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359" gracePeriod=30
Aug 01 00:45:50.910534 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:50.910488    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-tdr8r" podStartSLOduration=1.5897071170000001 podStartE2EDuration="4.910470802s" totalImagesPullingTime="3.320763685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:50.909543165 +0000 UTC m=+717.143475433" watchObservedRunningTime="2026-08-01 00:45:50.910470802 +0000 UTC m=+717.144403068"
Aug 01 00:45:51.131582 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.131558    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-tdr8r"
Aug 01 00:45:51.282476 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.282398    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz2cx\" (UniqueName: \"kubernetes.io/projected/8a895e7a-fde0-4946-b109-7f313e2f686a-kube-api-access-gz2cx\") pod \"8a895e7a-fde0-4946-b109-7f313e2f686a\" (UID: \"8a895e7a-fde0-4946-b109-7f313e2f686a\") "
Aug 01 00:45:51.284424 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.284393    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a895e7a-fde0-4946-b109-7f313e2f686a-kube-api-access-gz2cx" (OuterVolumeSpecName: "kube-api-access-gz2cx") pod "8a895e7a-fde0-4946-b109-7f313e2f686a" (UID: "8a895e7a-fde0-4946-b109-7f313e2f686a"). InnerVolumeSpecName "kube-api-access-gz2cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:45:51.383823 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.383791    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz2cx\" (UniqueName: \"kubernetes.io/projected/8a895e7a-fde0-4946-b109-7f313e2f686a-kube-api-access-gz2cx\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:45:51.902154 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.902111    2570 generic.go:356] "Generic (PLEG): container finished" podID="8a895e7a-fde0-4946-b109-7f313e2f686a" containerID="10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359" exitCode=0
Aug 01 00:45:51.902154 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.902158    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-tdr8r"
Aug 01 00:45:51.902667 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.902197    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-tdr8r" event={"ID":"8a895e7a-fde0-4946-b109-7f313e2f686a","Type":"ContainerDied","Data":"10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359"}
Aug 01 00:45:51.902667 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.902246    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-tdr8r" event={"ID":"8a895e7a-fde0-4946-b109-7f313e2f686a","Type":"ContainerDied","Data":"56ffdac19411ad9b65e464a5a6844b2949ab506f83e90bd3b5510d6a5af4a191"}
Aug 01 00:45:51.902667 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.902266    2570 scope.go:117] "RemoveContainer" containerID="10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359"
Aug 01 00:45:51.910431 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.910412    2570 scope.go:117] "RemoveContainer" containerID="10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359"
Aug 01 00:45:51.910687 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:45:51.910664    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359\": container with ID starting with 10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359 not found: ID does not exist" containerID="10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359"
Aug 01 00:45:51.910775 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.910692    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359"} err="failed to get container status \"10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359\": rpc error: code = NotFound desc = could not find container \"10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359\": container with ID starting with 10647c248947ed7d2b21c9f3443959bbbfeef6eb6bed0a0323aceb8fbd129359 not found: ID does not exist"
Aug 01 00:45:51.947441 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.947411    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tdr8r"]
Aug 01 00:45:51.950092 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:51.950068    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tdr8r"]
Aug 01 00:45:52.261170 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:52.261093    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a895e7a-fde0-4946-b109-7f313e2f686a" path="/var/lib/kubelet/pods/8a895e7a-fde0-4946-b109-7f313e2f686a/volumes"
Aug 01 00:45:58.886395 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:58.886361    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:45:59.443631 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:59.443592    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"]
Aug 01 00:45:59.443872 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:59.443846    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp" podUID="e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41" containerName="limitador" containerID="cri-o://9bbc7eb3f10543b61c41b289967eea6a9202c127e7ab1eb3c4386a8572a9f841" gracePeriod=30
Aug 01 00:45:59.944907 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:59.944876    2570 generic.go:356] "Generic (PLEG): container finished" podID="e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41" containerID="9bbc7eb3f10543b61c41b289967eea6a9202c127e7ab1eb3c4386a8572a9f841" exitCode=0
Aug 01 00:45:59.945339 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:45:59.944965    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp" event={"ID":"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41","Type":"ContainerDied","Data":"9bbc7eb3f10543b61c41b289967eea6a9202c127e7ab1eb3c4386a8572a9f841"}
Aug 01 00:46:00.384788 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.384764    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:46:00.562369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.562279    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55725\" (UniqueName: \"kubernetes.io/projected/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-kube-api-access-55725\") pod \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\" (UID: \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\") "
Aug 01 00:46:00.562369 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.562332    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-config-file\") pod \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\" (UID: \"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41\") "
Aug 01 00:46:00.562715 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.562689    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-config-file" (OuterVolumeSpecName: "config-file") pod "e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41" (UID: "e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:46:00.564432 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.564402    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-kube-api-access-55725" (OuterVolumeSpecName: "kube-api-access-55725") pod "e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41" (UID: "e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41"). InnerVolumeSpecName "kube-api-access-55725". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:00.663437 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.663403    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55725\" (UniqueName: \"kubernetes.io/projected/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-kube-api-access-55725\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:46:00.663437 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.663431    2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41-config-file\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:46:00.950353 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.950321    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"
Aug 01 00:46:00.950353 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.950338    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-wcmdp" event={"ID":"e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41","Type":"ContainerDied","Data":"4ea4274f1616767e54504c58ecb7fd83f5f86731e7dc3a14aeebe3b30862c3bc"}
Aug 01 00:46:00.950830 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.950403    2570 scope.go:117] "RemoveContainer" containerID="9bbc7eb3f10543b61c41b289967eea6a9202c127e7ab1eb3c4386a8572a9f841"
Aug 01 00:46:00.972592 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.972565    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"]
Aug 01 00:46:00.976051 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:00.976023    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wcmdp"]
Aug 01 00:46:02.261758 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:02.261726    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41" path="/var/lib/kubelet/pods/e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41/volumes"
Aug 01 00:46:05.334362 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.334328    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"]
Aug 01 00:46:05.334810 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.334608    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a895e7a-fde0-4946-b109-7f313e2f686a" containerName="authorino"
Aug 01 00:46:05.334810 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.334619    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a895e7a-fde0-4946-b109-7f313e2f686a" containerName="authorino"
Aug 01 00:46:05.334810 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.334634    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41" containerName="limitador"
Aug 01 00:46:05.334810 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.334640    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41" containerName="limitador"
Aug 01 00:46:05.334810 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.334696    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3dfd9f4-7999-41b7-a3e0-90dc7beb0b41" containerName="limitador"
Aug 01 00:46:05.334810 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.334702    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a895e7a-fde0-4946-b109-7f313e2f686a" containerName="authorino"
Aug 01 00:46:05.339037 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.338926    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:05.341704 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.341675    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Aug 01 00:46:05.341814 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.341775    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 01 00:46:05.341814 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.341804    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Aug 01 00:46:05.342752 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.342733    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lzzvl\""
Aug 01 00:46:05.345575 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.345552    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"]
Aug 01 00:46:05.504102 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.504064    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2a9e6955-3ad0-4eb5-876b-2ed51efc036d-data\") pod \"postgres-69fb4594cf-bfvwf\" (UID: \"2a9e6955-3ad0-4eb5-876b-2ed51efc036d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:05.504277 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.504123    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8rr\" (UniqueName: \"kubernetes.io/projected/2a9e6955-3ad0-4eb5-876b-2ed51efc036d-kube-api-access-2b8rr\") pod \"postgres-69fb4594cf-bfvwf\" (UID: \"2a9e6955-3ad0-4eb5-876b-2ed51efc036d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:05.605260 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.605171    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2a9e6955-3ad0-4eb5-876b-2ed51efc036d-data\") pod \"postgres-69fb4594cf-bfvwf\" (UID: \"2a9e6955-3ad0-4eb5-876b-2ed51efc036d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:05.605260 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.605217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8rr\" (UniqueName: \"kubernetes.io/projected/2a9e6955-3ad0-4eb5-876b-2ed51efc036d-kube-api-access-2b8rr\") pod \"postgres-69fb4594cf-bfvwf\" (UID: \"2a9e6955-3ad0-4eb5-876b-2ed51efc036d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:05.605610 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.605584    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2a9e6955-3ad0-4eb5-876b-2ed51efc036d-data\") pod \"postgres-69fb4594cf-bfvwf\" (UID: \"2a9e6955-3ad0-4eb5-876b-2ed51efc036d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:05.613347 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.613323    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8rr\" (UniqueName: \"kubernetes.io/projected/2a9e6955-3ad0-4eb5-876b-2ed51efc036d-kube-api-access-2b8rr\") pod \"postgres-69fb4594cf-bfvwf\" (UID: \"2a9e6955-3ad0-4eb5-876b-2ed51efc036d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:05.651285 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.651255    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:05.781588 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.781559    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"]
Aug 01 00:46:05.784259 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:46:05.784231    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9e6955_3ad0_4eb5_876b_2ed51efc036d.slice/crio-ecede717b997b3a5572766a758ec2a1c5e6fb67059127c51b1edfe93c20cc5fd WatchSource:0}: Error finding container ecede717b997b3a5572766a758ec2a1c5e6fb67059127c51b1edfe93c20cc5fd: Status 404 returned error can't find the container with id ecede717b997b3a5572766a758ec2a1c5e6fb67059127c51b1edfe93c20cc5fd
Aug 01 00:46:05.973315 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:05.973224    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf" event={"ID":"2a9e6955-3ad0-4eb5-876b-2ed51efc036d","Type":"ContainerStarted","Data":"ecede717b997b3a5572766a758ec2a1c5e6fb67059127c51b1edfe93c20cc5fd"}
Aug 01 00:46:11.000488 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:11.000449    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf" event={"ID":"2a9e6955-3ad0-4eb5-876b-2ed51efc036d","Type":"ContainerStarted","Data":"82f29ba61756abfb171a13d537cd9057091ef7dadd8759381ac7ce2084019785"}
Aug 01 00:46:11.000908 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:11.000604    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:11.015530 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:11.015486    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf" podStartSLOduration=1.3616659979999999 podStartE2EDuration="6.015471978s" totalImagesPullingTime="4.65380598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:46:11.014683703 +0000 UTC m=+737.248615970" watchObservedRunningTime="2026-08-01 00:46:11.015471978 +0000 UTC m=+737.249404244"
Aug 01 00:46:17.032580 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.032550    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bfvwf"
Aug 01 00:46:17.605732 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.605697    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9xt6j"]
Aug 01 00:46:17.610099 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.610080    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9xt6j"
Aug 01 00:46:17.612556 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.612530    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 01 00:46:17.612686 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.612670    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 01 00:46:17.613603 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.613587    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5lsdz\""
Aug 01 00:46:17.615816 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.615795    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9xt6j"]
Aug 01 00:46:17.697979 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.697917    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxjjh\" (UniqueName: \"kubernetes.io/projected/e2e475b2-601b-4d34-beb8-acb167b01d4b-kube-api-access-cxjjh\") pod \"authorino-5dc7ccb5f-9xt6j\" (UID: \"e2e475b2-601b-4d34-beb8-acb167b01d4b\") " pod="kuadrant-system/authorino-5dc7ccb5f-9xt6j"
Aug 01 00:46:17.698176 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.698156    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9xt6j"]
Aug 01 00:46:17.698349 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:46:17.698331    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cxjjh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-9xt6j" podUID="e2e475b2-601b-4d34-beb8-acb167b01d4b"
Aug 01 00:46:17.723598 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.723564    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-pflr8"]
Aug 01 00:46:17.726861 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.726840    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:17.729395 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.729374    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Aug 01 00:46:17.734264 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.734237    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-pflr8"]
Aug 01 00:46:17.799247 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.799205    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mdv\" (UniqueName: \"kubernetes.io/projected/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-kube-api-access-96mdv\") pod \"authorino-687bd45585-pflr8\" (UID: \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\") " pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:17.799419 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.799261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjjh\" (UniqueName: \"kubernetes.io/projected/e2e475b2-601b-4d34-beb8-acb167b01d4b-kube-api-access-cxjjh\") pod \"authorino-5dc7ccb5f-9xt6j\" (UID: \"e2e475b2-601b-4d34-beb8-acb167b01d4b\") " pod="kuadrant-system/authorino-5dc7ccb5f-9xt6j"
Aug 01 00:46:17.799419 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.799297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-tls-cert\") pod \"authorino-687bd45585-pflr8\" (UID: \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\") " pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:17.807151 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.807125    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjjh\" (UniqueName: \"kubernetes.io/projected/e2e475b2-601b-4d34-beb8-acb167b01d4b-kube-api-access-cxjjh\") pod \"authorino-5dc7ccb5f-9xt6j\" (UID: \"e2e475b2-601b-4d34-beb8-acb167b01d4b\") " pod="kuadrant-system/authorino-5dc7ccb5f-9xt6j"
Aug 01 00:46:17.843708 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.843677    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-pflr8"]
Aug 01 00:46:17.843916 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:46:17.843898    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-96mdv tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-pflr8" podUID="bfa66a8d-a6a6-4592-971f-2e4f0900dbfc"
Aug 01 00:46:17.870954 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.870876    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76d5859b45-xd5tw"]
Aug 01 00:46:17.874085 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.874071    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:46:17.881699 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.881678    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d5859b45-xd5tw"]
Aug 01 00:46:17.899678 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.899648    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgsz\" (UniqueName: \"kubernetes.io/projected/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-kube-api-access-5jgsz\") pod \"authorino-76d5859b45-xd5tw\" (UID: \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\") " pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:46:17.899780 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.899688    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mdv\" (UniqueName: \"kubernetes.io/projected/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-kube-api-access-96mdv\") pod \"authorino-687bd45585-pflr8\" (UID: \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\") " pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:17.899831 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.899811    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-tls-cert\") pod \"authorino-687bd45585-pflr8\" (UID: \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\") " pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:17.899919 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.899892    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-tls-cert\") pod \"authorino-76d5859b45-xd5tw\" (UID: \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\") " pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:46:17.902206 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.902190    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-tls-cert\") pod \"authorino-687bd45585-pflr8\" (UID: \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\") " pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:17.907154 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:17.907128    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mdv\" (UniqueName: \"kubernetes.io/projected/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-kube-api-access-96mdv\") pod \"authorino-687bd45585-pflr8\" (UID: \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\") " pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:18.000787 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.000751    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-tls-cert\") pod \"authorino-76d5859b45-xd5tw\" (UID: \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\") " pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:46:18.000787 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.000793    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgsz\" (UniqueName: \"kubernetes.io/projected/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-kube-api-access-5jgsz\") pod \"authorino-76d5859b45-xd5tw\" (UID: \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\") " pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:46:18.003174 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.003150    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-tls-cert\") pod \"authorino-76d5859b45-xd5tw\" (UID: \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\") " pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:46:18.009497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.009475    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgsz\" (UniqueName: \"kubernetes.io/projected/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-kube-api-access-5jgsz\") pod \"authorino-76d5859b45-xd5tw\" (UID: \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\") " pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:46:18.028310 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.028288    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9xt6j"
Aug 01 00:46:18.028404 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.028289    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:18.035223 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.035206    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9xt6j"
Aug 01 00:46:18.037779 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.037763    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:18.101849 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.101820    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-tls-cert\") pod \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\" (UID: \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\") "
Aug 01 00:46:18.102024 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.101856    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96mdv\" (UniqueName: \"kubernetes.io/projected/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-kube-api-access-96mdv\") pod \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\" (UID: \"bfa66a8d-a6a6-4592-971f-2e4f0900dbfc\") "
Aug 01 00:46:18.102024 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.101875    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxjjh\" (UniqueName: \"kubernetes.io/projected/e2e475b2-601b-4d34-beb8-acb167b01d4b-kube-api-access-cxjjh\") pod \"e2e475b2-601b-4d34-beb8-acb167b01d4b\" (UID: \"e2e475b2-601b-4d34-beb8-acb167b01d4b\") "
Aug 01 00:46:18.104069 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.104044    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "bfa66a8d-a6a6-4592-971f-2e4f0900dbfc" (UID: "bfa66a8d-a6a6-4592-971f-2e4f0900dbfc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:46:18.104164 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.104081    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-kube-api-access-96mdv" (OuterVolumeSpecName: "kube-api-access-96mdv") pod "bfa66a8d-a6a6-4592-971f-2e4f0900dbfc" (UID: "bfa66a8d-a6a6-4592-971f-2e4f0900dbfc"). InnerVolumeSpecName "kube-api-access-96mdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:18.104164 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.104117    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e475b2-601b-4d34-beb8-acb167b01d4b-kube-api-access-cxjjh" (OuterVolumeSpecName: "kube-api-access-cxjjh") pod "e2e475b2-601b-4d34-beb8-acb167b01d4b" (UID: "e2e475b2-601b-4d34-beb8-acb167b01d4b"). InnerVolumeSpecName "kube-api-access-cxjjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:18.184172 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.184092    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:46:18.202529 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.202501    2570 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-tls-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:46:18.202529 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.202529    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96mdv\" (UniqueName: \"kubernetes.io/projected/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc-kube-api-access-96mdv\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:46:18.202672 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.202540    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxjjh\" (UniqueName: \"kubernetes.io/projected/e2e475b2-601b-4d34-beb8-acb167b01d4b-kube-api-access-cxjjh\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:46:18.310247 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:18.310221    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d5859b45-xd5tw"]
Aug 01 00:46:18.312314 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:46:18.312287    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6e2c2a3_8e9a_4acb_8ad3_3d8caf9b5847.slice/crio-da3ebac4ffbef49beb66f6e49765bb1f95a8aae8042e3588c75cfa87d86111d8 WatchSource:0}: Error finding container da3ebac4ffbef49beb66f6e49765bb1f95a8aae8042e3588c75cfa87d86111d8: Status 404 returned error can't find the container with id da3ebac4ffbef49beb66f6e49765bb1f95a8aae8042e3588c75cfa87d86111d8
Aug 01 00:46:19.033602 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:19.033572    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-pflr8"
Aug 01 00:46:19.033770 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:19.033604    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d5859b45-xd5tw" event={"ID":"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847","Type":"ContainerStarted","Data":"da3ebac4ffbef49beb66f6e49765bb1f95a8aae8042e3588c75cfa87d86111d8"}
Aug 01 00:46:19.033770 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:19.033736    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-9xt6j"
Aug 01 00:46:19.059306 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:19.059245    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9xt6j"]
Aug 01 00:46:19.060983 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:19.060959    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-9xt6j"]
Aug 01 00:46:19.081076 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:19.081011    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-pflr8"]
Aug 01 00:46:19.087660 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:19.087632    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-pflr8"]
Aug 01 00:46:20.040109 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:20.040074    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d5859b45-xd5tw" event={"ID":"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847","Type":"ContainerStarted","Data":"eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a"}
Aug 01 00:46:20.056672 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:20.056619    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76d5859b45-xd5tw" podStartSLOduration=2.12092284 podStartE2EDuration="3.056603447s" totalImagesPullingTime="935.680607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:46:20.055532563 +0000 UTC m=+746.289464840" watchObservedRunningTime="2026-08-01 00:46:20.056603447 +0000 UTC m=+746.290535710"
Aug 01 00:46:20.261592 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:20.261558    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfa66a8d-a6a6-4592-971f-2e4f0900dbfc" path="/var/lib/kubelet/pods/bfa66a8d-a6a6-4592-971f-2e4f0900dbfc/volumes"
Aug 01 00:46:20.261967 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:20.261798    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e475b2-601b-4d34-beb8-acb167b01d4b" path="/var/lib/kubelet/pods/e2e475b2-601b-4d34-beb8-acb167b01d4b/volumes"
Aug 01 00:46:24.962835 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:24.962799    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787d5bc65-vksvk"]
Aug 01 00:46:24.969536 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:24.969515    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:24.972204 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:24.972180    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Aug 01 00:46:24.973190 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:24.973117    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ldrkx\""
Aug 01 00:46:24.973299 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:24.973246    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Aug 01 00:46:24.975500 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:24.975481    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787d5bc65-vksvk"]
Aug 01 00:46:25.060631 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:25.060599    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p6m4\" (UniqueName: \"kubernetes.io/projected/4f8a6a4f-2f4a-466b-8505-454501afcf19-kube-api-access-9p6m4\") pod \"maas-controller-787d5bc65-vksvk\" (UID: \"4f8a6a4f-2f4a-466b-8505-454501afcf19\") " pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:25.060801 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:25.060679    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6a4f-2f4a-466b-8505-454501afcf19-webhook-cert\") pod \"maas-controller-787d5bc65-vksvk\" (UID: \"4f8a6a4f-2f4a-466b-8505-454501afcf19\") " pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:25.161285 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:25.161251    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6m4\" (UniqueName: \"kubernetes.io/projected/4f8a6a4f-2f4a-466b-8505-454501afcf19-kube-api-access-9p6m4\") pod \"maas-controller-787d5bc65-vksvk\" (UID: \"4f8a6a4f-2f4a-466b-8505-454501afcf19\") " pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:25.161444 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:25.161353    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6a4f-2f4a-466b-8505-454501afcf19-webhook-cert\") pod \"maas-controller-787d5bc65-vksvk\" (UID: \"4f8a6a4f-2f4a-466b-8505-454501afcf19\") " pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:25.163708 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:25.163690    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6a4f-2f4a-466b-8505-454501afcf19-webhook-cert\") pod \"maas-controller-787d5bc65-vksvk\" (UID: \"4f8a6a4f-2f4a-466b-8505-454501afcf19\") " pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:25.170080 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:25.170061    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p6m4\" (UniqueName: \"kubernetes.io/projected/4f8a6a4f-2f4a-466b-8505-454501afcf19-kube-api-access-9p6m4\") pod \"maas-controller-787d5bc65-vksvk\" (UID: \"4f8a6a4f-2f4a-466b-8505-454501afcf19\") " pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:25.282023 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:25.281912    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:25.408047 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:25.408014    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787d5bc65-vksvk"]
Aug 01 00:46:25.411289 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:46:25.411255    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8a6a4f_2f4a_466b_8505_454501afcf19.slice/crio-516c4e50d3a55b6fe7eb07fda4be8f9b3aa9125aa3111957cd2af05871ac01fe WatchSource:0}: Error finding container 516c4e50d3a55b6fe7eb07fda4be8f9b3aa9125aa3111957cd2af05871ac01fe: Status 404 returned error can't find the container with id 516c4e50d3a55b6fe7eb07fda4be8f9b3aa9125aa3111957cd2af05871ac01fe
Aug 01 00:46:26.066182 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:26.066143    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-vksvk" event={"ID":"4f8a6a4f-2f4a-466b-8505-454501afcf19","Type":"ContainerStarted","Data":"516c4e50d3a55b6fe7eb07fda4be8f9b3aa9125aa3111957cd2af05871ac01fe"}
Aug 01 00:46:27.072392 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:27.072352    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-vksvk" event={"ID":"4f8a6a4f-2f4a-466b-8505-454501afcf19","Type":"ContainerStarted","Data":"8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e"}
Aug 01 00:46:27.072796 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:27.072484    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:27.092647 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:27.092600    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787d5bc65-vksvk" podStartSLOduration=1.6098291869999999 podStartE2EDuration="3.092584356s" totalImagesPullingTime="1.482755169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:46:27.091200599 +0000 UTC m=+753.325132877" watchObservedRunningTime="2026-08-01 00:46:27.092584356 +0000 UTC m=+753.326516621"
Aug 01 00:46:38.079595 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:38.079557    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:46:57.911175 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:57.911133    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"]
Aug 01 00:46:57.917589 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:57.917568    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:57.920407 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:57.920380    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Aug 01 00:46:57.921382 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:57.921366    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c4l5w\""
Aug 01 00:46:57.926305 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:57.925689    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"]
Aug 01 00:46:57.944904 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:57.944869    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-maas-api-tls\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:57.945068 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:57.944991    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnnz\" (UniqueName: \"kubernetes.io/projected/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-kube-api-access-mvnnz\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:57.945068 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:57.945016    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-openshift-service-ca\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:58.045983 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.045950    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnnz\" (UniqueName: \"kubernetes.io/projected/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-kube-api-access-mvnnz\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:58.045983 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.045989    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-openshift-service-ca\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:58.046209 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.046109    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-maas-api-tls\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:58.046706 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.046687    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-openshift-service-ca\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:58.048658 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.048635    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-maas-api-tls\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:58.054853 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.054831    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnnz\" (UniqueName: \"kubernetes.io/projected/40b2c1c2-f74a-412f-b796-2f0e80d5c22a-kube-api-access-mvnnz\") pod \"maas-api-8696875bbd-bnc6p\" (UID: \"40b2c1c2-f74a-412f-b796-2f0e80d5c22a\") " pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:58.229545 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.229451    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:46:58.367991 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.367961    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"]
Aug 01 00:46:58.371853 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:46:58.371814    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b2c1c2_f74a_412f_b796_2f0e80d5c22a.slice/crio-4f5a519aca6ed5983abb098d1d50770c5b9faac77b51626b1ad073a22b833e49 WatchSource:0}: Error finding container 4f5a519aca6ed5983abb098d1d50770c5b9faac77b51626b1ad073a22b833e49: Status 404 returned error can't find the container with id 4f5a519aca6ed5983abb098d1d50770c5b9faac77b51626b1ad073a22b833e49
Aug 01 00:46:58.509187 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.509108    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"]
Aug 01 00:46:58.514497 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.514480    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:46:58.517226 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.517204    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-prjrx\""
Aug 01 00:46:58.517345 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.517205    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Aug 01 00:46:58.523316 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.523294    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"]
Aug 01 00:46:58.549595 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.549567    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7t4s\" (UniqueName: \"kubernetes.io/projected/3020d525-e75f-4340-b6c2-e28a56c409b4-kube-api-access-b7t4s\") pod \"payload-pre-processing-687dbb6cf6-ld4rx\" (UID: \"3020d525-e75f-4340-b6c2-e28a56c409b4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:46:58.549721 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.549607    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3020d525-e75f-4340-b6c2-e28a56c409b4-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-ld4rx\" (UID: \"3020d525-e75f-4340-b6c2-e28a56c409b4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:46:58.650405 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.650370    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3020d525-e75f-4340-b6c2-e28a56c409b4-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-ld4rx\" (UID: \"3020d525-e75f-4340-b6c2-e28a56c409b4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:46:58.650575 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.650510    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7t4s\" (UniqueName: \"kubernetes.io/projected/3020d525-e75f-4340-b6c2-e28a56c409b4-kube-api-access-b7t4s\") pod \"payload-pre-processing-687dbb6cf6-ld4rx\" (UID: \"3020d525-e75f-4340-b6c2-e28a56c409b4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:46:58.651136 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.651117    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3020d525-e75f-4340-b6c2-e28a56c409b4-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-ld4rx\" (UID: \"3020d525-e75f-4340-b6c2-e28a56c409b4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:46:58.658892 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.658870    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7t4s\" (UniqueName: \"kubernetes.io/projected/3020d525-e75f-4340-b6c2-e28a56c409b4-kube-api-access-b7t4s\") pod \"payload-pre-processing-687dbb6cf6-ld4rx\" (UID: \"3020d525-e75f-4340-b6c2-e28a56c409b4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:46:58.826365 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.826328    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:46:58.951570 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:58.951540    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"]
Aug 01 00:46:58.954060 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:46:58.954032    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3020d525_e75f_4340_b6c2_e28a56c409b4.slice/crio-ffd7e2a624fd293907f76bae26a54a2fe51d5058f4bf581e9e34c76a8ecadebc WatchSource:0}: Error finding container ffd7e2a624fd293907f76bae26a54a2fe51d5058f4bf581e9e34c76a8ecadebc: Status 404 returned error can't find the container with id ffd7e2a624fd293907f76bae26a54a2fe51d5058f4bf581e9e34c76a8ecadebc
Aug 01 00:46:59.212142 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:59.212060    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx" event={"ID":"3020d525-e75f-4340-b6c2-e28a56c409b4","Type":"ContainerStarted","Data":"ffd7e2a624fd293907f76bae26a54a2fe51d5058f4bf581e9e34c76a8ecadebc"}
Aug 01 00:46:59.213462 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:46:59.213428    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p" event={"ID":"40b2c1c2-f74a-412f-b796-2f0e80d5c22a","Type":"ContainerStarted","Data":"4f5a519aca6ed5983abb098d1d50770c5b9faac77b51626b1ad073a22b833e49"}
Aug 01 00:47:00.220380 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:47:00.220290    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p" event={"ID":"40b2c1c2-f74a-412f-b796-2f0e80d5c22a","Type":"ContainerStarted","Data":"76381b16b3af2b6575b1e370fd318b068bee47f310512ce843c6cdc668695eba"}
Aug 01 00:47:00.220823 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:47:00.220417    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:47:00.238043 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:47:00.237992    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p" podStartSLOduration=1.695447895 podStartE2EDuration="3.237975953s" totalImagesPullingTime="1.542528058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:47:00.236542671 +0000 UTC m=+786.470474937" watchObservedRunningTime="2026-08-01 00:47:00.237975953 +0000 UTC m=+786.471908219"
Aug 01 00:47:03.239634 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:47:03.239595    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx" event={"ID":"3020d525-e75f-4340-b6c2-e28a56c409b4","Type":"ContainerStarted","Data":"5b12dd85ee074c4f60a6f3018aed1218632511dce9c2f5b547c3525018688684"}
Aug 01 00:47:03.240144 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:47:03.239680    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:47:03.255121 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:47:03.255070    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx" podStartSLOduration=2.042972443 podStartE2EDuration="5.255054457s" totalImagesPullingTime="3.212082014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:47:03.254115811 +0000 UTC m=+789.488048091" watchObservedRunningTime="2026-08-01 00:47:03.255054457 +0000 UTC m=+789.488986722"
Aug 01 00:47:06.231317 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:47:06.231287    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8696875bbd-bnc6p"
Aug 01 00:47:14.246842 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:47:14.246805    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ld4rx"
Aug 01 00:48:10.903529 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.903495    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"]
Aug 01 00:48:10.907182 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.907161    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:10.910147 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.910115    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Aug 01 00:48:10.910147 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.910135    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Aug 01 00:48:10.910326 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.910206    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h6wmw\""
Aug 01 00:48:10.910326 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.910231    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Aug 01 00:48:10.916822 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.916799    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"]
Aug 01 00:48:10.963550 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.963521    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:10.963704 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.963571    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:10.963704 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.963680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eacbf814-ce99-403e-84d8-756772bcfd53-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:10.963819 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.963794    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:10.963861 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.963820    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:10.963861 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:10.963844    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcm6\" (UniqueName: \"kubernetes.io/projected/eacbf814-ce99-403e-84d8-756772bcfd53-kube-api-access-bwcm6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.064470 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.064435    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.064470 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.064470    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.064687 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.064494    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eacbf814-ce99-403e-84d8-756772bcfd53-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.064687 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.064531    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.064687 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.064548    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.064687 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.064573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcm6\" (UniqueName: \"kubernetes.io/projected/eacbf814-ce99-403e-84d8-756772bcfd53-kube-api-access-bwcm6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.064974 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.064913    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.064974 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.064960    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.065154 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.065048    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.066926 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.066903    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eacbf814-ce99-403e-84d8-756772bcfd53-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.067562 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.067539    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eacbf814-ce99-403e-84d8-756772bcfd53-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.072418 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.072398    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcm6\" (UniqueName: \"kubernetes.io/projected/eacbf814-ce99-403e-84d8-756772bcfd53-kube-api-access-bwcm6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dxzgn\" (UID: \"eacbf814-ce99-403e-84d8-756772bcfd53\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.219655 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.219554    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:11.351739 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.351716    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"]
Aug 01 00:48:11.354282 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:48:11.354254    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacbf814_ce99_403e_84d8_756772bcfd53.slice/crio-aed321377014a30418e27e922c92b7487238b2da9b1a991fe1996cb87f711f95 WatchSource:0}: Error finding container aed321377014a30418e27e922c92b7487238b2da9b1a991fe1996cb87f711f95: Status 404 returned error can't find the container with id aed321377014a30418e27e922c92b7487238b2da9b1a991fe1996cb87f711f95
Aug 01 00:48:11.356070 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.356054    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:48:11.538651 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:11.538578    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn" event={"ID":"eacbf814-ce99-403e-84d8-756772bcfd53","Type":"ContainerStarted","Data":"aed321377014a30418e27e922c92b7487238b2da9b1a991fe1996cb87f711f95"}
Aug 01 00:48:12.544530 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:12.544500    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn" event={"ID":"eacbf814-ce99-403e-84d8-756772bcfd53","Type":"ContainerStarted","Data":"659e267522ade5073b3641964e8e391a33380612ae30adb332ff136508b94f13"}
Aug 01 00:48:12.563159 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:12.563100    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn" podStartSLOduration=1.461619292 podStartE2EDuration="2.563081806s" totalImagesPullingTime="1.101462514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:48:12.562227003 +0000 UTC m=+858.796159271" watchObservedRunningTime="2026-08-01 00:48:12.563081806 +0000 UTC m=+858.797014075"
Aug 01 00:48:21.219980 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:21.219919    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:21.220394 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:21.219999    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:21.233948 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:21.233910    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:48:21.597101 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:48:21.597072    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dxzgn"
Aug 01 00:50:10.052104 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.052068    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d78f965b-b2dv4"]
Aug 01 00:50:10.055473 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.055454    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d78f965b-b2dv4"
Aug 01 00:50:10.064624 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.064599    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d78f965b-b2dv4"]
Aug 01 00:50:10.185349 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.185318    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49034aed-cac8-43c0-bb75-8674a201d17e-tls-cert\") pod \"authorino-75d78f965b-b2dv4\" (UID: \"49034aed-cac8-43c0-bb75-8674a201d17e\") " pod="kuadrant-system/authorino-75d78f965b-b2dv4"
Aug 01 00:50:10.185548 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.185435    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5r4w\" (UniqueName: \"kubernetes.io/projected/49034aed-cac8-43c0-bb75-8674a201d17e-kube-api-access-r5r4w\") pod \"authorino-75d78f965b-b2dv4\" (UID: \"49034aed-cac8-43c0-bb75-8674a201d17e\") " pod="kuadrant-system/authorino-75d78f965b-b2dv4"
Aug 01 00:50:10.286800 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.286764    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5r4w\" (UniqueName: \"kubernetes.io/projected/49034aed-cac8-43c0-bb75-8674a201d17e-kube-api-access-r5r4w\") pod \"authorino-75d78f965b-b2dv4\" (UID: \"49034aed-cac8-43c0-bb75-8674a201d17e\") " pod="kuadrant-system/authorino-75d78f965b-b2dv4"
Aug 01 00:50:10.287041 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.286827    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49034aed-cac8-43c0-bb75-8674a201d17e-tls-cert\") pod \"authorino-75d78f965b-b2dv4\" (UID: \"49034aed-cac8-43c0-bb75-8674a201d17e\") " pod="kuadrant-system/authorino-75d78f965b-b2dv4"
Aug 01 00:50:10.289243 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.289221    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49034aed-cac8-43c0-bb75-8674a201d17e-tls-cert\") pod \"authorino-75d78f965b-b2dv4\" (UID: \"49034aed-cac8-43c0-bb75-8674a201d17e\") " pod="kuadrant-system/authorino-75d78f965b-b2dv4"
Aug 01 00:50:10.294945 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.294914    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5r4w\" (UniqueName: \"kubernetes.io/projected/49034aed-cac8-43c0-bb75-8674a201d17e-kube-api-access-r5r4w\") pod \"authorino-75d78f965b-b2dv4\" (UID: \"49034aed-cac8-43c0-bb75-8674a201d17e\") " pod="kuadrant-system/authorino-75d78f965b-b2dv4"
Aug 01 00:50:10.365625 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.365585    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d78f965b-b2dv4"
Aug 01 00:50:10.498627 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:10.498600    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d78f965b-b2dv4"]
Aug 01 00:50:10.500868 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:50:10.500836    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49034aed_cac8_43c0_bb75_8674a201d17e.slice/crio-8fd4928afe936bdd04b3b185da8004e25e488ddba74bd12aba1a339a77ae9ab5 WatchSource:0}: Error finding container 8fd4928afe936bdd04b3b185da8004e25e488ddba74bd12aba1a339a77ae9ab5: Status 404 returned error can't find the container with id 8fd4928afe936bdd04b3b185da8004e25e488ddba74bd12aba1a339a77ae9ab5
Aug 01 00:50:11.073686 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:11.073651    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d78f965b-b2dv4" event={"ID":"49034aed-cac8-43c0-bb75-8674a201d17e","Type":"ContainerStarted","Data":"8fd4928afe936bdd04b3b185da8004e25e488ddba74bd12aba1a339a77ae9ab5"}
Aug 01 00:50:12.082017 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.081975    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d78f965b-b2dv4" event={"ID":"49034aed-cac8-43c0-bb75-8674a201d17e","Type":"ContainerStarted","Data":"860e83f9bb9d4586fd1132c3b595e383e3b8fc5094c95c83ca3f62a407f0d3b4"}
Aug 01 00:50:12.101534 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.101460    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d78f965b-b2dv4" podStartSLOduration=1.21796357 podStartE2EDuration="2.101438252s" totalImagesPullingTime="883.474682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:50:12.100592808 +0000 UTC m=+978.334525074" watchObservedRunningTime="2026-08-01 00:50:12.101438252 +0000 UTC m=+978.335370520"
Aug 01 00:50:12.134835 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.134786    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d5859b45-xd5tw"]
Aug 01 00:50:12.135511 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.135126    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-76d5859b45-xd5tw" podUID="b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847" containerName="authorino" containerID="cri-o://eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a" gracePeriod=30
Aug 01 00:50:12.419248 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.419218    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:50:12.508560 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.508520    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgsz\" (UniqueName: \"kubernetes.io/projected/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-kube-api-access-5jgsz\") pod \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\" (UID: \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\") "
Aug 01 00:50:12.508749 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.508639    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-tls-cert\") pod \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\" (UID: \"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847\") "
Aug 01 00:50:12.510677 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.510647    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-kube-api-access-5jgsz" (OuterVolumeSpecName: "kube-api-access-5jgsz") pod "b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847" (UID: "b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847"). InnerVolumeSpecName "kube-api-access-5jgsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:50:12.518683 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.518657    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847" (UID: "b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:50:12.610155 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.610120    2570 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-tls-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:50:12.610155 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:12.610154    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgsz\" (UniqueName: \"kubernetes.io/projected/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847-kube-api-access-5jgsz\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:50:13.089429 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.089394    2570 generic.go:356] "Generic (PLEG): container finished" podID="b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847" containerID="eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a" exitCode=0
Aug 01 00:50:13.089859 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.089439    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d5859b45-xd5tw"
Aug 01 00:50:13.089859 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.089485    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d5859b45-xd5tw" event={"ID":"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847","Type":"ContainerDied","Data":"eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a"}
Aug 01 00:50:13.089859 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.089525    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d5859b45-xd5tw" event={"ID":"b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847","Type":"ContainerDied","Data":"da3ebac4ffbef49beb66f6e49765bb1f95a8aae8042e3588c75cfa87d86111d8"}
Aug 01 00:50:13.089859 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.089547    2570 scope.go:117] "RemoveContainer" containerID="eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a"
Aug 01 00:50:13.099925 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.099907    2570 scope.go:117] "RemoveContainer" containerID="eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a"
Aug 01 00:50:13.100216 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:50:13.100189    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a\": container with ID starting with eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a not found: ID does not exist" containerID="eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a"
Aug 01 00:50:13.100319 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.100224    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a"} err="failed to get container status \"eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a\": rpc error: code = NotFound desc = could not find container \"eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a\": container with ID starting with eddb0eaa8dde2f9161d51574ad58df8394c463123dda4c1cb38bab77d1a30a7a not found: ID does not exist"
Aug 01 00:50:13.110652 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.110624    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d5859b45-xd5tw"]
Aug 01 00:50:13.113852 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:13.113830    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76d5859b45-xd5tw"]
Aug 01 00:50:14.261742 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:50:14.261707    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847" path="/var/lib/kubelet/pods/b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847/volumes"
Aug 01 00:51:00.339321 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.339282    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787d5bc65-vksvk"]
Aug 01 00:51:00.339784 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.339527    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-787d5bc65-vksvk" podUID="4f8a6a4f-2f4a-466b-8505-454501afcf19" containerName="manager" containerID="cri-o://8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e" gracePeriod=30
Aug 01 00:51:00.584711 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.584685    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:51:00.711190 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.711092    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p6m4\" (UniqueName: \"kubernetes.io/projected/4f8a6a4f-2f4a-466b-8505-454501afcf19-kube-api-access-9p6m4\") pod \"4f8a6a4f-2f4a-466b-8505-454501afcf19\" (UID: \"4f8a6a4f-2f4a-466b-8505-454501afcf19\") "
Aug 01 00:51:00.711190 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.711165    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6a4f-2f4a-466b-8505-454501afcf19-webhook-cert\") pod \"4f8a6a4f-2f4a-466b-8505-454501afcf19\" (UID: \"4f8a6a4f-2f4a-466b-8505-454501afcf19\") "
Aug 01 00:51:00.713383 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.713353    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8a6a4f-2f4a-466b-8505-454501afcf19-kube-api-access-9p6m4" (OuterVolumeSpecName: "kube-api-access-9p6m4") pod "4f8a6a4f-2f4a-466b-8505-454501afcf19" (UID: "4f8a6a4f-2f4a-466b-8505-454501afcf19"). InnerVolumeSpecName "kube-api-access-9p6m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:51:00.713503 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.713403    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8a6a4f-2f4a-466b-8505-454501afcf19-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4f8a6a4f-2f4a-466b-8505-454501afcf19" (UID: "4f8a6a4f-2f4a-466b-8505-454501afcf19"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:51:00.812161 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.812128    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p6m4\" (UniqueName: \"kubernetes.io/projected/4f8a6a4f-2f4a-466b-8505-454501afcf19-kube-api-access-9p6m4\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:51:00.812161 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:00.812155    2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6a4f-2f4a-466b-8505-454501afcf19-webhook-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:51:01.300419 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.300384    2570 generic.go:356] "Generic (PLEG): container finished" podID="4f8a6a4f-2f4a-466b-8505-454501afcf19" containerID="8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e" exitCode=0
Aug 01 00:51:01.300596 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.300426    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-vksvk"
Aug 01 00:51:01.300596 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.300473    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-vksvk" event={"ID":"4f8a6a4f-2f4a-466b-8505-454501afcf19","Type":"ContainerDied","Data":"8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e"}
Aug 01 00:51:01.300596 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.300534    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-vksvk" event={"ID":"4f8a6a4f-2f4a-466b-8505-454501afcf19","Type":"ContainerDied","Data":"516c4e50d3a55b6fe7eb07fda4be8f9b3aa9125aa3111957cd2af05871ac01fe"}
Aug 01 00:51:01.300596 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.300559    2570 scope.go:117] "RemoveContainer" containerID="8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e"
Aug 01 00:51:01.309288 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.309267    2570 scope.go:117] "RemoveContainer" containerID="8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e"
Aug 01 00:51:01.309543 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:51:01.309518    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e\": container with ID starting with 8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e not found: ID does not exist" containerID="8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e"
Aug 01 00:51:01.309623 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.309550    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e"} err="failed to get container status \"8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e\": rpc error: code = NotFound desc = could not find container \"8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e\": container with ID starting with 8e544d7f551d34eca1d4078ce340e7fd25ae1cfbdd848abcff651c69e8b6e42e not found: ID does not exist"
Aug 01 00:51:01.321627 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.321604    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787d5bc65-vksvk"]
Aug 01 00:51:01.324948 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:01.324910    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-787d5bc65-vksvk"]
Aug 01 00:51:02.262210 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:51:02.262175    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8a6a4f-2f4a-466b-8505-454501afcf19" path="/var/lib/kubelet/pods/4f8a6a4f-2f4a-466b-8505-454501afcf19/volumes"
Aug 01 00:52:02.169082 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.169045    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787d5bc65-sh8vq"]
Aug 01 00:52:02.169502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.169376    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847" containerName="authorino"
Aug 01 00:52:02.169502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.169387    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847" containerName="authorino"
Aug 01 00:52:02.169502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.169400    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8a6a4f-2f4a-466b-8505-454501afcf19" containerName="manager"
Aug 01 00:52:02.169502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.169406    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8a6a4f-2f4a-466b-8505-454501afcf19" containerName="manager"
Aug 01 00:52:02.169502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.169476    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8a6a4f-2f4a-466b-8505-454501afcf19" containerName="manager"
Aug 01 00:52:02.169502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.169485    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e2c2a3-8e9a-4acb-8ad3-3d8caf9b5847" containerName="authorino"
Aug 01 00:52:02.173653 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.173635    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:02.177081 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.177059    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ldrkx\""
Aug 01 00:52:02.177213 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.177087    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Aug 01 00:52:02.177213 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.177190    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Aug 01 00:52:02.184869 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.184849    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787d5bc65-sh8vq"]
Aug 01 00:52:02.344240 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.344199    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8fg\" (UniqueName: \"kubernetes.io/projected/92f78703-acc7-4463-988b-63e8584f3268-kube-api-access-kg8fg\") pod \"maas-controller-787d5bc65-sh8vq\" (UID: \"92f78703-acc7-4463-988b-63e8584f3268\") " pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:02.344435 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.344288    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92f78703-acc7-4463-988b-63e8584f3268-webhook-cert\") pod \"maas-controller-787d5bc65-sh8vq\" (UID: \"92f78703-acc7-4463-988b-63e8584f3268\") " pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:02.445215 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.445134    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92f78703-acc7-4463-988b-63e8584f3268-webhook-cert\") pod \"maas-controller-787d5bc65-sh8vq\" (UID: \"92f78703-acc7-4463-988b-63e8584f3268\") " pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:02.445215 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.445194    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8fg\" (UniqueName: \"kubernetes.io/projected/92f78703-acc7-4463-988b-63e8584f3268-kube-api-access-kg8fg\") pod \"maas-controller-787d5bc65-sh8vq\" (UID: \"92f78703-acc7-4463-988b-63e8584f3268\") " pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:02.447597 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.447573    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92f78703-acc7-4463-988b-63e8584f3268-webhook-cert\") pod \"maas-controller-787d5bc65-sh8vq\" (UID: \"92f78703-acc7-4463-988b-63e8584f3268\") " pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:02.453427 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.453402    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8fg\" (UniqueName: \"kubernetes.io/projected/92f78703-acc7-4463-988b-63e8584f3268-kube-api-access-kg8fg\") pod \"maas-controller-787d5bc65-sh8vq\" (UID: \"92f78703-acc7-4463-988b-63e8584f3268\") " pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:02.483943 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.483898    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:02.618695 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:02.618667    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787d5bc65-sh8vq"]
Aug 01 00:52:02.621287 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:52:02.621255    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f78703_acc7_4463_988b_63e8584f3268.slice/crio-df7841a588db126f02979cbe0554b9202c6391be91b17d89d987b39f19a89607 WatchSource:0}: Error finding container df7841a588db126f02979cbe0554b9202c6391be91b17d89d987b39f19a89607: Status 404 returned error can't find the container with id df7841a588db126f02979cbe0554b9202c6391be91b17d89d987b39f19a89607
Aug 01 00:52:03.582532 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:03.582487    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-sh8vq" event={"ID":"92f78703-acc7-4463-988b-63e8584f3268","Type":"ContainerStarted","Data":"9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a"}
Aug 01 00:52:03.582992 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:03.582690    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:03.582992 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:03.582734    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-sh8vq" event={"ID":"92f78703-acc7-4463-988b-63e8584f3268","Type":"ContainerStarted","Data":"df7841a588db126f02979cbe0554b9202c6391be91b17d89d987b39f19a89607"}
Aug 01 00:52:03.600086 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:03.600036    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787d5bc65-sh8vq" podStartSLOduration=0.990884919 podStartE2EDuration="1.600021838s" totalImagesPullingTime="609.136919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:52:03.599796567 +0000 UTC m=+1089.833728834" watchObservedRunningTime="2026-08-01 00:52:03.600021838 +0000 UTC m=+1089.833954104"
Aug 01 00:52:14.589620 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:14.589587    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:19.236615 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.236579    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787d5bc65-sh8vq"]
Aug 01 00:52:19.237083 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.236801    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-787d5bc65-sh8vq" podUID="92f78703-acc7-4463-988b-63e8584f3268" containerName="manager" containerID="cri-o://9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a" gracePeriod=30
Aug 01 00:52:19.487889 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.487829    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:19.590402 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.590369    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92f78703-acc7-4463-988b-63e8584f3268-webhook-cert\") pod \"92f78703-acc7-4463-988b-63e8584f3268\" (UID: \"92f78703-acc7-4463-988b-63e8584f3268\") "
Aug 01 00:52:19.590611 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.590436    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg8fg\" (UniqueName: \"kubernetes.io/projected/92f78703-acc7-4463-988b-63e8584f3268-kube-api-access-kg8fg\") pod \"92f78703-acc7-4463-988b-63e8584f3268\" (UID: \"92f78703-acc7-4463-988b-63e8584f3268\") "
Aug 01 00:52:19.592548 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.592510    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f78703-acc7-4463-988b-63e8584f3268-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "92f78703-acc7-4463-988b-63e8584f3268" (UID: "92f78703-acc7-4463-988b-63e8584f3268"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:52:19.592548 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.592510    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f78703-acc7-4463-988b-63e8584f3268-kube-api-access-kg8fg" (OuterVolumeSpecName: "kube-api-access-kg8fg") pod "92f78703-acc7-4463-988b-63e8584f3268" (UID: "92f78703-acc7-4463-988b-63e8584f3268"). InnerVolumeSpecName "kube-api-access-kg8fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:52:19.654145 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.654113    2570 generic.go:356] "Generic (PLEG): container finished" podID="92f78703-acc7-4463-988b-63e8584f3268" containerID="9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a" exitCode=0
Aug 01 00:52:19.654315 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.654161    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-sh8vq"
Aug 01 00:52:19.654315 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.654183    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-sh8vq" event={"ID":"92f78703-acc7-4463-988b-63e8584f3268","Type":"ContainerDied","Data":"9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a"}
Aug 01 00:52:19.654315 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.654224    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-sh8vq" event={"ID":"92f78703-acc7-4463-988b-63e8584f3268","Type":"ContainerDied","Data":"df7841a588db126f02979cbe0554b9202c6391be91b17d89d987b39f19a89607"}
Aug 01 00:52:19.654315 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.654245    2570 scope.go:117] "RemoveContainer" containerID="9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a"
Aug 01 00:52:19.663114 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.663098    2570 scope.go:117] "RemoveContainer" containerID="9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a"
Aug 01 00:52:19.663371 ip-10-0-141-142 kubenswrapper[2570]: E0801 00:52:19.663348    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a\": container with ID starting with 9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a not found: ID does not exist" containerID="9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a"
Aug 01 00:52:19.663435 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.663382    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a"} err="failed to get container status \"9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a\": rpc error: code = NotFound desc = could not find container \"9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a\": container with ID starting with 9342e039b1b81626f9d824326790f8fa4781ed33b23036e481da6ce0543f556a not found: ID does not exist"
Aug 01 00:52:19.674332 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.674307    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-787d5bc65-sh8vq"]
Aug 01 00:52:19.678403 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.678380    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-787d5bc65-sh8vq"]
Aug 01 00:52:19.691203 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.691185    2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92f78703-acc7-4463-988b-63e8584f3268-webhook-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:52:19.691276 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:19.691203    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg8fg\" (UniqueName: \"kubernetes.io/projected/92f78703-acc7-4463-988b-63e8584f3268-kube-api-access-kg8fg\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 00:52:20.262201 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.262170    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92f78703-acc7-4463-988b-63e8584f3268" path="/var/lib/kubelet/pods/92f78703-acc7-4463-988b-63e8584f3268/volumes"
Aug 01 00:52:20.501806 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.501772    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-787d5bc65-p4m2m"]
Aug 01 00:52:20.502147 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.502133    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f78703-acc7-4463-988b-63e8584f3268" containerName="manager"
Aug 01 00:52:20.502203 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.502148    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f78703-acc7-4463-988b-63e8584f3268" containerName="manager"
Aug 01 00:52:20.502203 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.502198    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f78703-acc7-4463-988b-63e8584f3268" containerName="manager"
Aug 01 00:52:20.506386 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.506370    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:20.508947 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.508916    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Aug 01 00:52:20.509872 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.509855    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Aug 01 00:52:20.509976 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.509890    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ldrkx\""
Aug 01 00:52:20.512696 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.512634    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787d5bc65-p4m2m"]
Aug 01 00:52:20.599381 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.599342    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5778d33b-8872-4191-96ab-c210256d3c25-webhook-cert\") pod \"maas-controller-787d5bc65-p4m2m\" (UID: \"5778d33b-8872-4191-96ab-c210256d3c25\") " pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:20.599565 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.599458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjd4b\" (UniqueName: \"kubernetes.io/projected/5778d33b-8872-4191-96ab-c210256d3c25-kube-api-access-kjd4b\") pod \"maas-controller-787d5bc65-p4m2m\" (UID: \"5778d33b-8872-4191-96ab-c210256d3c25\") " pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:20.700156 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.700121    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5778d33b-8872-4191-96ab-c210256d3c25-webhook-cert\") pod \"maas-controller-787d5bc65-p4m2m\" (UID: \"5778d33b-8872-4191-96ab-c210256d3c25\") " pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:20.700329 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.700185    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjd4b\" (UniqueName: \"kubernetes.io/projected/5778d33b-8872-4191-96ab-c210256d3c25-kube-api-access-kjd4b\") pod \"maas-controller-787d5bc65-p4m2m\" (UID: \"5778d33b-8872-4191-96ab-c210256d3c25\") " pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:20.702612 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.702591    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5778d33b-8872-4191-96ab-c210256d3c25-webhook-cert\") pod \"maas-controller-787d5bc65-p4m2m\" (UID: \"5778d33b-8872-4191-96ab-c210256d3c25\") " pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:20.707693 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.707666    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjd4b\" (UniqueName: \"kubernetes.io/projected/5778d33b-8872-4191-96ab-c210256d3c25-kube-api-access-kjd4b\") pod \"maas-controller-787d5bc65-p4m2m\" (UID: \"5778d33b-8872-4191-96ab-c210256d3c25\") " pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:20.817639 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.817610    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:20.944502 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:20.944470    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-787d5bc65-p4m2m"]
Aug 01 00:52:20.948096 ip-10-0-141-142 kubenswrapper[2570]: W0801 00:52:20.948065    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5778d33b_8872_4191_96ab_c210256d3c25.slice/crio-fec5ab1f9c7d5658ff2e341146e8a8fede671b0e62f4237f480f9a9dc4d80dc9 WatchSource:0}: Error finding container fec5ab1f9c7d5658ff2e341146e8a8fede671b0e62f4237f480f9a9dc4d80dc9: Status 404 returned error can't find the container with id fec5ab1f9c7d5658ff2e341146e8a8fede671b0e62f4237f480f9a9dc4d80dc9
Aug 01 00:52:21.667210 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:21.667177    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-p4m2m" event={"ID":"5778d33b-8872-4191-96ab-c210256d3c25","Type":"ContainerStarted","Data":"5e85c66e719dd21af20df6d6ac079fd586a2353eaf61de22632620d649dc1157"}
Aug 01 00:52:21.667667 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:21.667238    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-787d5bc65-p4m2m" event={"ID":"5778d33b-8872-4191-96ab-c210256d3c25","Type":"ContainerStarted","Data":"fec5ab1f9c7d5658ff2e341146e8a8fede671b0e62f4237f480f9a9dc4d80dc9"}
Aug 01 00:52:21.667667 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:21.667282    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 00:52:21.683946 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:21.683892    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-787d5bc65-p4m2m" podStartSLOduration=1.364030332 podStartE2EDuration="1.683875851s" totalImagesPullingTime="319.845519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:52:21.682640353 +0000 UTC m=+1107.916572621" watchObservedRunningTime="2026-08-01 00:52:21.683875851 +0000 UTC m=+1107.917808147"
Aug 01 00:52:32.674317 ip-10-0-141-142 kubenswrapper[2570]: I0801 00:52:32.674285    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-787d5bc65-p4m2m"
Aug 01 01:00:00.141702 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:00.141665    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx"]
Aug 01 01:00:00.144854 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:00.144838    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx"
Aug 01 01:00:00.147502 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:00.147484    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\""
Aug 01 01:00:00.152422 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:00.152368    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx"]
Aug 01 01:00:00.155511 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:00.155491    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx"
Aug 01 01:00:00.289724 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:00.289698    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx"]
Aug 01 01:00:00.291722 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:00:00.291690    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb011ff59_9da2_448e_9350_98b7e1b4699f.slice/crio-5a96cdf0d7dfc4eddc913c71040cac38fa076cfe53a09dfbac60af3ac3ce32ae WatchSource:0}: Error finding container 5a96cdf0d7dfc4eddc913c71040cac38fa076cfe53a09dfbac60af3ac3ce32ae: Status 404 returned error can't find the container with id 5a96cdf0d7dfc4eddc913c71040cac38fa076cfe53a09dfbac60af3ac3ce32ae
Aug 01 01:00:00.293599 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:00.293583    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 01:00:00.712158 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:00.712121    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx" event={"ID":"b011ff59-9da2-448e-9350-98b7e1b4699f","Type":"ContainerStarted","Data":"5a96cdf0d7dfc4eddc913c71040cac38fa076cfe53a09dfbac60af3ac3ce32ae"}
Aug 01 01:00:03.728450 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:03.728365    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx" event={"ID":"b011ff59-9da2-448e-9350-98b7e1b4699f","Type":"ContainerStarted","Data":"aba8c3dcb3368efc125d88cb5440dda061571fdaf6969804b3ad6900d6758ccb"}
Aug 01 01:00:03.746429 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:03.746379    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx" podStartSLOduration=0.82969636 podStartE2EDuration="3.746364061s" totalImagesPullingTime="2.916667701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:00:03.744867232 +0000 UTC m=+1569.978799497" watchObservedRunningTime="2026-08-01 01:00:03.746364061 +0000 UTC m=+1569.980296327"
Aug 01 01:00:04.734622 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:04.734584    2570 generic.go:356] "Generic (PLEG): container finished" podID="b011ff59-9da2-448e-9350-98b7e1b4699f" containerID="aba8c3dcb3368efc125d88cb5440dda061571fdaf6969804b3ad6900d6758ccb" exitCode=0
Aug 01 01:00:04.735037 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:04.734642    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx" event={"ID":"b011ff59-9da2-448e-9350-98b7e1b4699f","Type":"ContainerDied","Data":"aba8c3dcb3368efc125d88cb5440dda061571fdaf6969804b3ad6900d6758ccb"}
Aug 01 01:00:05.871907 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:05.871879    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx"
Aug 01 01:00:06.744677 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:06.744645    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx"
Aug 01 01:00:06.744867 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:06.744649    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759100-68zjx" event={"ID":"b011ff59-9da2-448e-9350-98b7e1b4699f","Type":"ContainerDied","Data":"5a96cdf0d7dfc4eddc913c71040cac38fa076cfe53a09dfbac60af3ac3ce32ae"}
Aug 01 01:00:06.744867 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:00:06.744784    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a96cdf0d7dfc4eddc913c71040cac38fa076cfe53a09dfbac60af3ac3ce32ae"
Aug 01 01:02:36.996147 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:36.996068    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"]
Aug 01 01:02:36.996589 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:36.996470    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b011ff59-9da2-448e-9350-98b7e1b4699f" containerName="cleanup"
Aug 01 01:02:36.996589 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:36.996484    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b011ff59-9da2-448e-9350-98b7e1b4699f" containerName="cleanup"
Aug 01 01:02:36.996589 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:36.996534    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b011ff59-9da2-448e-9350-98b7e1b4699f" containerName="cleanup"
Aug 01 01:02:36.999369 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:36.999348    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.001790 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.001767    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2283368e-kserve-self-signed-certs\""
Aug 01 01:02:37.013368 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.013344    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"]
Aug 01 01:02:37.100462 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.100426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-home\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.100462 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.100460    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-dshm\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.100663 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.100476    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-model-cache\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.100663 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.100550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96zx\" (UniqueName: \"kubernetes.io/projected/e632fec1-2316-4db7-9edb-0ca0d081679b-kube-api-access-s96zx\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.100663 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.100581    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e632fec1-2316-4db7-9edb-0ca0d081679b-tls-certs\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.100768 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.100670    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-tmp-dir\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.201785 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.201756    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-tmp-dir\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.201924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.201810    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-home\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.201924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.201824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-dshm\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.201924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.201840    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-model-cache\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.201924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.201876    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96zx\" (UniqueName: \"kubernetes.io/projected/e632fec1-2316-4db7-9edb-0ca0d081679b-kube-api-access-s96zx\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.201924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.201895    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e632fec1-2316-4db7-9edb-0ca0d081679b-tls-certs\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.202235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.202208    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-tmp-dir\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.202316 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.202254    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-home\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.202369 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.202315    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-model-cache\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.204068 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.204051    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-dshm\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.204416 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.204399    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e632fec1-2316-4db7-9edb-0ca0d081679b-tls-certs\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.209751 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.209730    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96zx\" (UniqueName: \"kubernetes.io/projected/e632fec1-2316-4db7-9edb-0ca0d081679b-kube-api-access-s96zx\") pod \"e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") " pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.310035 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.309979    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:37.439964 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:37.439924    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"]
Aug 01 01:02:37.441604 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:02:37.441575    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode632fec1_2316_4db7_9edb_0ca0d081679b.slice/crio-a09fed98bce38207cfe12d0e451d66869a60fd48a960c1d445a2ab8441873c51 WatchSource:0}: Error finding container a09fed98bce38207cfe12d0e451d66869a60fd48a960c1d445a2ab8441873c51: Status 404 returned error can't find the container with id a09fed98bce38207cfe12d0e451d66869a60fd48a960c1d445a2ab8441873c51
Aug 01 01:02:38.420767 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:38.420726    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j" event={"ID":"e632fec1-2316-4db7-9edb-0ca0d081679b","Type":"ContainerStarted","Data":"a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b"}
Aug 01 01:02:38.421207 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:38.420780    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j" event={"ID":"e632fec1-2316-4db7-9edb-0ca0d081679b","Type":"ContainerStarted","Data":"a09fed98bce38207cfe12d0e451d66869a60fd48a960c1d445a2ab8441873c51"}
Aug 01 01:02:38.436273 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:38.436194    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j" podStartSLOduration=2.436177288 podStartE2EDuration="2.436177288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:02:38.435529633 +0000 UTC m=+1724.669461901" watchObservedRunningTime="2026-08-01 01:02:38.436177288 +0000 UTC m=+1724.670109554"
Aug 01 01:02:47.310247 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:47.310164    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:47.310247 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:47.310212    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:47.322571 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:47.322547    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:47.478151 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:47.478124    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:02:51.895197 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:51.895161    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"]
Aug 01 01:02:51.898711 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:51.898690    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:51.901604 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:51.901577    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2283368e-kserve-self-signed-certs\""
Aug 01 01:02:51.912061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:51.912036    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"]
Aug 01 01:02:52.022168 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.022127    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-dshm\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.022331 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.022189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0355a2-4223-45c0-a3bb-81408feed333-tls-certs\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.022331 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.022261    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55nh\" (UniqueName: \"kubernetes.io/projected/1c0355a2-4223-45c0-a3bb-81408feed333-kube-api-access-s55nh\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.022331 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.022292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-model-cache\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.022475 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.022339    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-home\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.022475 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.022367    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-tmp-dir\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.123480 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.123446    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s55nh\" (UniqueName: \"kubernetes.io/projected/1c0355a2-4223-45c0-a3bb-81408feed333-kube-api-access-s55nh\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.123619 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.123487    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-model-cache\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.123619 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.123534    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-home\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.123619 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.123560    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-tmp-dir\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.123619 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.123589    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-dshm\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.123849 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.123628    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0355a2-4223-45c0-a3bb-81408feed333-tls-certs\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.124004 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.123980    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-model-cache\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.124079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.124059    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-tmp-dir\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.124122 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.124073    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-home\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.125801 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.125779    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-dshm\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.126038 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.126021    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0355a2-4223-45c0-a3bb-81408feed333-tls-certs\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.131024 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.131000    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55nh\" (UniqueName: \"kubernetes.io/projected/1c0355a2-4223-45c0-a3bb-81408feed333-kube-api-access-s55nh\") pod \"e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") " pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.215783 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.215699    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:02:52.342585 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.342554    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"]
Aug 01 01:02:52.345714 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:02:52.345682    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c0355a2_4223_45c0_a3bb_81408feed333.slice/crio-7f029716a0c2d3b0fc286b6a9f467c816e6e7d282ccb780c26284ae09762e90e WatchSource:0}: Error finding container 7f029716a0c2d3b0fc286b6a9f467c816e6e7d282ccb780c26284ae09762e90e: Status 404 returned error can't find the container with id 7f029716a0c2d3b0fc286b6a9f467c816e6e7d282ccb780c26284ae09762e90e
Aug 01 01:02:52.489865 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.489774    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx" event={"ID":"1c0355a2-4223-45c0-a3bb-81408feed333","Type":"ContainerStarted","Data":"585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0"}
Aug 01 01:02:52.489865 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.489825    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx" event={"ID":"1c0355a2-4223-45c0-a3bb-81408feed333","Type":"ContainerStarted","Data":"7f029716a0c2d3b0fc286b6a9f467c816e6e7d282ccb780c26284ae09762e90e"}
Aug 01 01:02:52.506252 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:02:52.506198    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx" podStartSLOduration=1.5061787899999999 podStartE2EDuration="1.50617879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:02:52.505849053 +0000 UTC m=+1738.739781319" watchObservedRunningTime="2026-08-01 01:02:52.50617879 +0000 UTC m=+1738.740111069"
Aug 01 01:03:02.216786 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:02.216741    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:03:02.217285 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:02.216957    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:03:02.229422 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:02.229396    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:03:02.544708 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:02.544618    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:03:26.763203 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:26.763166    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"]
Aug 01 01:03:27.581278 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:27.581246    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"]
Aug 01 01:03:41.784542 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:41.784482    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j" podUID="e632fec1-2316-4db7-9edb-0ca0d081679b" containerName="main" containerID="cri-o://a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b" gracePeriod=45
Aug 01 01:03:42.039781 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.039724    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:03:42.102428 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102403    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-model-cache\") pod \"e632fec1-2316-4db7-9edb-0ca0d081679b\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") "
Aug 01 01:03:42.102563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102442    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-dshm\") pod \"e632fec1-2316-4db7-9edb-0ca0d081679b\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") "
Aug 01 01:03:42.102563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102497    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s96zx\" (UniqueName: \"kubernetes.io/projected/e632fec1-2316-4db7-9edb-0ca0d081679b-kube-api-access-s96zx\") pod \"e632fec1-2316-4db7-9edb-0ca0d081679b\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") "
Aug 01 01:03:42.102563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102513    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-tmp-dir\") pod \"e632fec1-2316-4db7-9edb-0ca0d081679b\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") "
Aug 01 01:03:42.102563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102549    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-home\") pod \"e632fec1-2316-4db7-9edb-0ca0d081679b\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") "
Aug 01 01:03:42.102774 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102579    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e632fec1-2316-4db7-9edb-0ca0d081679b-tls-certs\") pod \"e632fec1-2316-4db7-9edb-0ca0d081679b\" (UID: \"e632fec1-2316-4db7-9edb-0ca0d081679b\") "
Aug 01 01:03:42.102774 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102678    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e632fec1-2316-4db7-9edb-0ca0d081679b" (UID: "e632fec1-2316-4db7-9edb-0ca0d081679b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:03:42.102774 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102770    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e632fec1-2316-4db7-9edb-0ca0d081679b" (UID: "e632fec1-2316-4db7-9edb-0ca0d081679b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:03:42.102948 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102817    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-home" (OuterVolumeSpecName: "home") pod "e632fec1-2316-4db7-9edb-0ca0d081679b" (UID: "e632fec1-2316-4db7-9edb-0ca0d081679b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:03:42.102948 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102845    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:42.102948 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.102856    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:42.104720 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.104694    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-dshm" (OuterVolumeSpecName: "dshm") pod "e632fec1-2316-4db7-9edb-0ca0d081679b" (UID: "e632fec1-2316-4db7-9edb-0ca0d081679b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:03:42.104817 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.104785    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e632fec1-2316-4db7-9edb-0ca0d081679b-kube-api-access-s96zx" (OuterVolumeSpecName: "kube-api-access-s96zx") pod "e632fec1-2316-4db7-9edb-0ca0d081679b" (UID: "e632fec1-2316-4db7-9edb-0ca0d081679b"). InnerVolumeSpecName "kube-api-access-s96zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:03:42.104858 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.104838    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e632fec1-2316-4db7-9edb-0ca0d081679b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e632fec1-2316-4db7-9edb-0ca0d081679b" (UID: "e632fec1-2316-4db7-9edb-0ca0d081679b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:03:42.203348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.203318    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s96zx\" (UniqueName: \"kubernetes.io/projected/e632fec1-2316-4db7-9edb-0ca0d081679b-kube-api-access-s96zx\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:42.203348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.203344    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:42.203348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.203354    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e632fec1-2316-4db7-9edb-0ca0d081679b-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:42.203541 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.203363    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e632fec1-2316-4db7-9edb-0ca0d081679b-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:42.600717 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.600680    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx" podUID="1c0355a2-4223-45c0-a3bb-81408feed333" containerName="main" containerID="cri-o://585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0" gracePeriod=45
Aug 01 01:03:42.714903 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.714869    2570 generic.go:356] "Generic (PLEG): container finished" podID="e632fec1-2316-4db7-9edb-0ca0d081679b" containerID="a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b" exitCode=0
Aug 01 01:03:42.715077 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.714917    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"
Aug 01 01:03:42.715077 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.714965    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j" event={"ID":"e632fec1-2316-4db7-9edb-0ca0d081679b","Type":"ContainerDied","Data":"a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b"}
Aug 01 01:03:42.715077 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.715010    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j" event={"ID":"e632fec1-2316-4db7-9edb-0ca0d081679b","Type":"ContainerDied","Data":"a09fed98bce38207cfe12d0e451d66869a60fd48a960c1d445a2ab8441873c51"}
Aug 01 01:03:42.715077 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.715027    2570 scope.go:117] "RemoveContainer" containerID="a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b"
Aug 01 01:03:42.723622 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.723601    2570 scope.go:117] "RemoveContainer" containerID="a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b"
Aug 01 01:03:42.723876 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:03:42.723854    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b\": container with ID starting with a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b not found: ID does not exist" containerID="a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b"
Aug 01 01:03:42.723965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.723883    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b"} err="failed to get container status \"a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b\": rpc error: code = NotFound desc = could not find container \"a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b\": container with ID starting with a39fe39ca1aa0bed14b4063ed2de594b223025487de3e6389447fcea02e90b6b not found: ID does not exist"
Aug 01 01:03:42.731264 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.731241    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"]
Aug 01 01:03:42.737206 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.737186    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2283368e-kserve-7c6cb9c6c5-xqr8j"]
Aug 01 01:03:42.851389 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.851341    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:03:42.911690 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.911659    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-model-cache\") pod \"1c0355a2-4223-45c0-a3bb-81408feed333\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") "
Aug 01 01:03:42.911805 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.911720    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-home\") pod \"1c0355a2-4223-45c0-a3bb-81408feed333\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") "
Aug 01 01:03:42.911805 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.911754    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0355a2-4223-45c0-a3bb-81408feed333-tls-certs\") pod \"1c0355a2-4223-45c0-a3bb-81408feed333\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") "
Aug 01 01:03:42.911805 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.911776    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-dshm\") pod \"1c0355a2-4223-45c0-a3bb-81408feed333\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") "
Aug 01 01:03:42.911805 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.911794    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-tmp-dir\") pod \"1c0355a2-4223-45c0-a3bb-81408feed333\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") "
Aug 01 01:03:42.912023 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.911814    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s55nh\" (UniqueName: \"kubernetes.io/projected/1c0355a2-4223-45c0-a3bb-81408feed333-kube-api-access-s55nh\") pod \"1c0355a2-4223-45c0-a3bb-81408feed333\" (UID: \"1c0355a2-4223-45c0-a3bb-81408feed333\") "
Aug 01 01:03:42.912023 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.911922    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c0355a2-4223-45c0-a3bb-81408feed333" (UID: "1c0355a2-4223-45c0-a3bb-81408feed333"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:03:42.912130 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.912027    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:42.912130 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.912053    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-home" (OuterVolumeSpecName: "home") pod "1c0355a2-4223-45c0-a3bb-81408feed333" (UID: "1c0355a2-4223-45c0-a3bb-81408feed333"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:03:42.912130 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.912076    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c0355a2-4223-45c0-a3bb-81408feed333" (UID: "1c0355a2-4223-45c0-a3bb-81408feed333"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:03:42.913813 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.913790    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0355a2-4223-45c0-a3bb-81408feed333-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c0355a2-4223-45c0-a3bb-81408feed333" (UID: "1c0355a2-4223-45c0-a3bb-81408feed333"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:03:42.913813 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.913798    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0355a2-4223-45c0-a3bb-81408feed333-kube-api-access-s55nh" (OuterVolumeSpecName: "kube-api-access-s55nh") pod "1c0355a2-4223-45c0-a3bb-81408feed333" (UID: "1c0355a2-4223-45c0-a3bb-81408feed333"). InnerVolumeSpecName "kube-api-access-s55nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:03:42.913914 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:42.913872    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-dshm" (OuterVolumeSpecName: "dshm") pod "1c0355a2-4223-45c0-a3bb-81408feed333" (UID: "1c0355a2-4223-45c0-a3bb-81408feed333"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:03:43.012783 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.012758    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:43.012783 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.012780    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0355a2-4223-45c0-a3bb-81408feed333-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:43.012919 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.012790    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:43.012919 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.012816    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0355a2-4223-45c0-a3bb-81408feed333-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:43.012919 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.012825    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s55nh\" (UniqueName: \"kubernetes.io/projected/1c0355a2-4223-45c0-a3bb-81408feed333-kube-api-access-s55nh\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:03:43.720878 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.720847    2570 generic.go:356] "Generic (PLEG): container finished" podID="1c0355a2-4223-45c0-a3bb-81408feed333" containerID="585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0" exitCode=0
Aug 01 01:03:43.721063 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.720892    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"
Aug 01 01:03:43.721063 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.720896    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx" event={"ID":"1c0355a2-4223-45c0-a3bb-81408feed333","Type":"ContainerDied","Data":"585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0"}
Aug 01 01:03:43.721063 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.721029    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx" event={"ID":"1c0355a2-4223-45c0-a3bb-81408feed333","Type":"ContainerDied","Data":"7f029716a0c2d3b0fc286b6a9f467c816e6e7d282ccb780c26284ae09762e90e"}
Aug 01 01:03:43.721063 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.721059    2570 scope.go:117] "RemoveContainer" containerID="585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0"
Aug 01 01:03:43.730046 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.730029    2570 scope.go:117] "RemoveContainer" containerID="585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0"
Aug 01 01:03:43.730284 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:03:43.730264    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0\": container with ID starting with 585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0 not found: ID does not exist" containerID="585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0"
Aug 01 01:03:43.730322 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.730295    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0"} err="failed to get container status \"585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0\": rpc error: code = NotFound desc = could not find container \"585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0\": container with ID starting with 585b18e1529cd1e914f589506d55ac7f32e0a510ecdde1b696b9accf93b2a7a0 not found: ID does not exist"
Aug 01 01:03:43.743629 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.743604    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"]
Aug 01 01:03:43.746867 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:43.746843    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2283368e-kserve-76794b6fcf-zhtlx"]
Aug 01 01:03:44.261428 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:44.261398    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0355a2-4223-45c0-a3bb-81408feed333" path="/var/lib/kubelet/pods/1c0355a2-4223-45c0-a3bb-81408feed333/volumes"
Aug 01 01:03:44.261797 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:03:44.261784    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e632fec1-2316-4db7-9edb-0ca0d081679b" path="/var/lib/kubelet/pods/e632fec1-2316-4db7-9edb-0ca0d081679b/volumes"
Aug 01 01:07:38.782992 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.782959    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"]
Aug 01 01:07:38.783447 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.783285    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e632fec1-2316-4db7-9edb-0ca0d081679b" containerName="main"
Aug 01 01:07:38.783447 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.783296    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e632fec1-2316-4db7-9edb-0ca0d081679b" containerName="main"
Aug 01 01:07:38.783447 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.783321    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0355a2-4223-45c0-a3bb-81408feed333" containerName="main"
Aug 01 01:07:38.783447 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.783326    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0355a2-4223-45c0-a3bb-81408feed333" containerName="main"
Aug 01 01:07:38.783447 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.783376    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0355a2-4223-45c0-a3bb-81408feed333" containerName="main"
Aug 01 01:07:38.783447 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.783385    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e632fec1-2316-4db7-9edb-0ca0d081679b" containerName="main"
Aug 01 01:07:38.786304 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.786287    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.789043 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.789015    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7ed58d7f-openshift-default-dockercfg-24zwn\""
Aug 01 01:07:38.789177 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.789090    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Aug 01 01:07:38.808982 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.808956    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"]
Aug 01 01:07:38.816326 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816293    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-certs\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.816419 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816360    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-data\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.816464 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816419    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-credential-socket\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.816464 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816439    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-podinfo\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.816543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816478    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-socket\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.816543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816512    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-envoy\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.816617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816545    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-token\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.816617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816591    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istiod-ca-cert\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.816617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.816606    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psn6c\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-kube-api-access-psn6c\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917289 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917258    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-envoy\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917413 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-token\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917413 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917314    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istiod-ca-cert\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917505 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917423    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psn6c\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-kube-api-access-psn6c\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917505 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917466    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-certs\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917608 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917501    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-data\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917608 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917558    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-credential-socket\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917608 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917587    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-podinfo\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917748 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917653    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-socket\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.917896 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917873    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-certs\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.918020 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.917997    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-credential-socket\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.918103 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.918009    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-data\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.918154 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.918096    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-socket\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.918154 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.918114    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istiod-ca-cert\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.919603 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.919581    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-envoy\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.919972 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.919954    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-podinfo\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.925446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.925420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-token\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:38.925589 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:38.925567    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psn6c\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-kube-api-access-psn6c\") pod \"e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") " pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:39.096744 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:39.096713    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:39.230079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:39.230051    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"]
Aug 01 01:07:39.231614 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:07:39.231589    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ea2abe_cb61_4f2d_848c_7c94ba7b9238.slice/crio-aa913ddf27aa410f25ec318103e2185be778103612af570d45f960ef46bbd639 WatchSource:0}: Error finding container aa913ddf27aa410f25ec318103e2185be778103612af570d45f960ef46bbd639: Status 404 returned error can't find the container with id aa913ddf27aa410f25ec318103e2185be778103612af570d45f960ef46bbd639
Aug 01 01:07:39.233550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:39.233523    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 01:07:39.804618 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:39.804580    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" event={"ID":"67ea2abe-cb61-4f2d-848c-7c94ba7b9238","Type":"ContainerStarted","Data":"aa913ddf27aa410f25ec318103e2185be778103612af570d45f960ef46bbd639"}
Aug 01 01:07:42.996112 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:42.996078    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:07:42.996364 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:42.996149    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:07:42.996364 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:42.996179    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:07:43.824162 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:43.824125    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" event={"ID":"67ea2abe-cb61-4f2d-848c-7c94ba7b9238","Type":"ContainerStarted","Data":"d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841"}
Aug 01 01:07:43.844693 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:43.844638    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" podStartSLOduration=2.082518203 podStartE2EDuration="5.844624882s" totalImagesPullingTime="3.762106679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:07:43.843376938 +0000 UTC m=+2030.077309204" watchObservedRunningTime="2026-08-01 01:07:43.844624882 +0000 UTC m=+2030.078557147"
Aug 01 01:07:44.097720 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:44.097644    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:44.102538 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:44.102516    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:44.828733 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:44.828702    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:44.829581 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:44.829564    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:46.408093 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.408062    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"]
Aug 01 01:07:46.411604 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.411584    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.414125 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.414103    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7ed58d7f-serving-cert\""
Aug 01 01:07:46.422400 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.422376    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"]
Aug 01 01:07:46.480380 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.480356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.480494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.480422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbdzf\" (UniqueName: \"kubernetes.io/projected/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-kube-api-access-gbdzf\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.480494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.480469    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-openshift-service-ca\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.581158 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.581133    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbdzf\" (UniqueName: \"kubernetes.io/projected/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-kube-api-access-gbdzf\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.581286 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.581185    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-openshift-service-ca\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.581286 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.581212    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.581356 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:07:46.581327    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-serving-cert: secret "maas-api-e2e-ait-7ed58d7f-serving-cert" not found
Aug 01 01:07:46.581393 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:07:46.581375    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls podName:4668aa5a-4cfe-4c76-b722-e06ec81d10d2 nodeName:}" failed. No retries permitted until 2026-08-01 01:07:47.081360294 +0000 UTC m=+2033.315292538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls") pod "maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v" (UID: "4668aa5a-4cfe-4c76-b722-e06ec81d10d2") : secret "maas-api-e2e-ait-7ed58d7f-serving-cert" not found
Aug 01 01:07:46.581825 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.581804    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-openshift-service-ca\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.589718 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.589698    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbdzf\" (UniqueName: \"kubernetes.io/projected/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-kube-api-access-gbdzf\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:46.977228 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.977193    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"]
Aug 01 01:07:46.980866 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.980842    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:46.983513 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.983493    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7ed58d7f-dockercfg-4wp77\""
Aug 01 01:07:46.983610 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.983520    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7ed58d7f\""
Aug 01 01:07:46.991309 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:46.991290    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"]
Aug 01 01:07:47.009073 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.009046    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"]
Aug 01 01:07:47.012478 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.012458    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.022911 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.022891    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"]
Aug 01 01:07:47.086958 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.086846    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:47.087168 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.087002    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58c7\" (UniqueName: \"kubernetes.io/projected/1c9c5760-831f-48e7-9987-571d2718ef8d-kube-api-access-d58c7\") pod \"payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t\" (UID: \"1c9c5760-831f-48e7-9987-571d2718ef8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:47.087168 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.087066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klxx8\" (UniqueName: \"kubernetes.io/projected/55dc7aa1-7895-4280-93a5-3042c475cbcb-kube-api-access-klxx8\") pod \"payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x\" (UID: \"55dc7aa1-7895-4280-93a5-3042c475cbcb\") " pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.087168 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.087119    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dc7aa1-7895-4280-93a5-3042c475cbcb-plugins-config-volume\") pod \"payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x\" (UID: \"55dc7aa1-7895-4280-93a5-3042c475cbcb\") " pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.087300 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.087175    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9c5760-831f-48e7-9987-571d2718ef8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t\" (UID: \"1c9c5760-831f-48e7-9987-571d2718ef8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:47.089536 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.089512    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls\") pod \"maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:47.187824 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.187787    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58c7\" (UniqueName: \"kubernetes.io/projected/1c9c5760-831f-48e7-9987-571d2718ef8d-kube-api-access-d58c7\") pod \"payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t\" (UID: \"1c9c5760-831f-48e7-9987-571d2718ef8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:47.188029 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.187846    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klxx8\" (UniqueName: \"kubernetes.io/projected/55dc7aa1-7895-4280-93a5-3042c475cbcb-kube-api-access-klxx8\") pod \"payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x\" (UID: \"55dc7aa1-7895-4280-93a5-3042c475cbcb\") " pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.188029 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.187875    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dc7aa1-7895-4280-93a5-3042c475cbcb-plugins-config-volume\") pod \"payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x\" (UID: \"55dc7aa1-7895-4280-93a5-3042c475cbcb\") " pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.188029 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.187963    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9c5760-831f-48e7-9987-571d2718ef8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t\" (UID: \"1c9c5760-831f-48e7-9987-571d2718ef8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:47.188657 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.188630    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dc7aa1-7895-4280-93a5-3042c475cbcb-plugins-config-volume\") pod \"payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x\" (UID: \"55dc7aa1-7895-4280-93a5-3042c475cbcb\") " pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.188760 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.188655    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9c5760-831f-48e7-9987-571d2718ef8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t\" (UID: \"1c9c5760-831f-48e7-9987-571d2718ef8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:47.195696 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.195659    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58c7\" (UniqueName: \"kubernetes.io/projected/1c9c5760-831f-48e7-9987-571d2718ef8d-kube-api-access-d58c7\") pod \"payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t\" (UID: \"1c9c5760-831f-48e7-9987-571d2718ef8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:47.195696 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.195671    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klxx8\" (UniqueName: \"kubernetes.io/projected/55dc7aa1-7895-4280-93a5-3042c475cbcb-kube-api-access-klxx8\") pod \"payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x\" (UID: \"55dc7aa1-7895-4280-93a5-3042c475cbcb\") " pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.292701 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.292634    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:47.323275 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.323249    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:47.324333 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.324315    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.466267 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.466218    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"]
Aug 01 01:07:47.469655 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:07:47.469624    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9c5760_831f_48e7_9987_571d2718ef8d.slice/crio-ea4e8fd5a0ec72484cd16915fc312c04c2b7bb496d3973323dc9402e5755d762 WatchSource:0}: Error finding container ea4e8fd5a0ec72484cd16915fc312c04c2b7bb496d3973323dc9402e5755d762: Status 404 returned error can't find the container with id ea4e8fd5a0ec72484cd16915fc312c04c2b7bb496d3973323dc9402e5755d762
Aug 01 01:07:47.697700 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.697676    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"]
Aug 01 01:07:47.700129 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:07:47.700099    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4668aa5a_4cfe_4c76_b722_e06ec81d10d2.slice/crio-4891bf69f83cc7df53106c076faaadc191fd50358a8eef71aac6a224913c1921 WatchSource:0}: Error finding container 4891bf69f83cc7df53106c076faaadc191fd50358a8eef71aac6a224913c1921: Status 404 returned error can't find the container with id 4891bf69f83cc7df53106c076faaadc191fd50358a8eef71aac6a224913c1921
Aug 01 01:07:47.702246 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.702221    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"]
Aug 01 01:07:47.703292 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:07:47.703267    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dc7aa1_7895_4280_93a5_3042c475cbcb.slice/crio-724d6bd116a3773a0bd251532c4a40e18fcb6650ffd4ea3697e57cfdf5dd958a WatchSource:0}: Error finding container 724d6bd116a3773a0bd251532c4a40e18fcb6650ffd4ea3697e57cfdf5dd958a: Status 404 returned error can't find the container with id 724d6bd116a3773a0bd251532c4a40e18fcb6650ffd4ea3697e57cfdf5dd958a
Aug 01 01:07:47.843596 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.843563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v" event={"ID":"4668aa5a-4cfe-4c76-b722-e06ec81d10d2","Type":"ContainerStarted","Data":"4891bf69f83cc7df53106c076faaadc191fd50358a8eef71aac6a224913c1921"}
Aug 01 01:07:47.845150 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.845115    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x" event={"ID":"55dc7aa1-7895-4280-93a5-3042c475cbcb","Type":"ContainerStarted","Data":"23789886248b0492d6db3fbd8a79b2a04dc14d5246e303807856f7dccc9b7a8c"}
Aug 01 01:07:47.845296 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.845176    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x" event={"ID":"55dc7aa1-7895-4280-93a5-3042c475cbcb","Type":"ContainerStarted","Data":"724d6bd116a3773a0bd251532c4a40e18fcb6650ffd4ea3697e57cfdf5dd958a"}
Aug 01 01:07:47.845364 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.845289    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:47.846615 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.846590    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t" event={"ID":"1c9c5760-831f-48e7-9987-571d2718ef8d","Type":"ContainerStarted","Data":"eacbb2494c2fee4c83bd0b1bdf7a7c3c5ca7a41f4d421e3728e1b68dacab8c90"}
Aug 01 01:07:47.846727 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.846634    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:47.846727 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.846650    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t" event={"ID":"1c9c5760-831f-48e7-9987-571d2718ef8d","Type":"ContainerStarted","Data":"ea4e8fd5a0ec72484cd16915fc312c04c2b7bb496d3973323dc9402e5755d762"}
Aug 01 01:07:47.861682 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.861637    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x" podStartSLOduration=1.861625273 podStartE2EDuration="1.861625273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:07:47.860647578 +0000 UTC m=+2034.094579847" watchObservedRunningTime="2026-08-01 01:07:47.861625273 +0000 UTC m=+2034.095557537"
Aug 01 01:07:47.876494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:47.876447    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t" podStartSLOduration=1.876434305 podStartE2EDuration="1.876434305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:07:47.875091501 +0000 UTC m=+2034.109023766" watchObservedRunningTime="2026-08-01 01:07:47.876434305 +0000 UTC m=+2034.110366571"
Aug 01 01:07:48.856122 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:48.856084    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v" event={"ID":"4668aa5a-4cfe-4c76-b722-e06ec81d10d2","Type":"ContainerStarted","Data":"8475bf16bc8054d0a7dae5acedf92b31154bc3c93f451615a13f52e4c1609df3"}
Aug 01 01:07:48.873127 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:48.873083    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v" podStartSLOduration=2.320029258 podStartE2EDuration="2.873069082s" totalImagesPullingTime="553.039824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:07:48.871877795 +0000 UTC m=+2035.105810063" watchObservedRunningTime="2026-08-01 01:07:48.873069082 +0000 UTC m=+2035.107001376"
Aug 01 01:07:49.861986 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:49.861957    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:55.875225 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:55.875199    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:07:58.858432 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:58.858362    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:07:58.858880 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:58.858438    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:07:58.912663 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:58.912629    2570 generic.go:356] "Generic (PLEG): container finished" podID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerID="d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841" exitCode=0
Aug 01 01:07:58.912803 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:58.912681    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" event={"ID":"67ea2abe-cb61-4f2d-848c-7c94ba7b9238","Type":"ContainerDied","Data":"d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841"}
Aug 01 01:07:58.913091 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:58.913073    2570 scope.go:117] "RemoveContainer" containerID="d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841"
Aug 01 01:07:58.914146 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:58.914121    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:07:58.914222 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:58.914184    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:07:58.914222 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:58.914215    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:07:59.828831 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:59.828787    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:07:59.920415 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:07:59.920375    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" event={"ID":"67ea2abe-cb61-4f2d-848c-7c94ba7b9238","Type":"ContainerStarted","Data":"071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088"}
Aug 01 01:08:00.097893 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:00.097806    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:08:00.098842 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:00.098810    2570 patch_prober.go:28] interesting pod/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Aug 01 01:08:00.099002 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:00.098885    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Aug 01 01:08:01.097885 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:01.097848    2570 patch_prober.go:28] interesting pod/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Aug 01 01:08:01.098298 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:01.097952    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Aug 01 01:08:02.098042 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:02.098007    2570 patch_prober.go:28] interesting pod/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Aug 01 01:08:02.098628 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:02.098069    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Aug 01 01:08:03.097708 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:03.097671    2570 patch_prober.go:28] interesting pod/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Aug 01 01:08:03.097887 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:03.097731    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Aug 01 01:08:04.097878 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:04.097841    2570 patch_prober.go:28] interesting pod/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Aug 01 01:08:04.098294 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:04.097903    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Aug 01 01:08:05.100945 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:05.100884    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:08:05.101380 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:05.101360    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:08:05.101991 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:05.101973    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:08:26.815603 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:26.815409    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"]
Aug 01 01:08:26.816214 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:26.815736    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v" podUID="4668aa5a-4cfe-4c76-b722-e06ec81d10d2" containerName="maas-api" containerID="cri-o://8475bf16bc8054d0a7dae5acedf92b31154bc3c93f451615a13f52e4c1609df3" gracePeriod=30
Aug 01 01:08:26.865548 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:26.865518    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"]
Aug 01 01:08:26.865782 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:26.865737    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x" podUID="55dc7aa1-7895-4280-93a5-3042c475cbcb" containerName="payload-processing" containerID="cri-o://23789886248b0492d6db3fbd8a79b2a04dc14d5246e303807856f7dccc9b7a8c" gracePeriod=30
Aug 01 01:08:26.871924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:26.871898    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"]
Aug 01 01:08:26.872145 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:26.872117    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t" podUID="1c9c5760-831f-48e7-9987-571d2718ef8d" containerName="payload-pre-processing" containerID="cri-o://eacbb2494c2fee4c83bd0b1bdf7a7c3c5ca7a41f4d421e3728e1b68dacab8c90" gracePeriod=30
Aug 01 01:08:27.054953 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.054781    2570 generic.go:356] "Generic (PLEG): container finished" podID="1c9c5760-831f-48e7-9987-571d2718ef8d" containerID="eacbb2494c2fee4c83bd0b1bdf7a7c3c5ca7a41f4d421e3728e1b68dacab8c90" exitCode=0
Aug 01 01:08:27.054953 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.054920    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t" event={"ID":"1c9c5760-831f-48e7-9987-571d2718ef8d","Type":"ContainerDied","Data":"eacbb2494c2fee4c83bd0b1bdf7a7c3c5ca7a41f4d421e3728e1b68dacab8c90"}
Aug 01 01:08:27.056661 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.056638    2570 generic.go:356] "Generic (PLEG): container finished" podID="4668aa5a-4cfe-4c76-b722-e06ec81d10d2" containerID="8475bf16bc8054d0a7dae5acedf92b31154bc3c93f451615a13f52e4c1609df3" exitCode=0
Aug 01 01:08:27.056752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.056712    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v" event={"ID":"4668aa5a-4cfe-4c76-b722-e06ec81d10d2","Type":"ContainerDied","Data":"8475bf16bc8054d0a7dae5acedf92b31154bc3c93f451615a13f52e4c1609df3"}
Aug 01 01:08:27.058232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.058212    2570 generic.go:356] "Generic (PLEG): container finished" podID="55dc7aa1-7895-4280-93a5-3042c475cbcb" containerID="23789886248b0492d6db3fbd8a79b2a04dc14d5246e303807856f7dccc9b7a8c" exitCode=0
Aug 01 01:08:27.058305 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.058258    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x" event={"ID":"55dc7aa1-7895-4280-93a5-3042c475cbcb","Type":"ContainerDied","Data":"23789886248b0492d6db3fbd8a79b2a04dc14d5246e303807856f7dccc9b7a8c"}
Aug 01 01:08:27.135097 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.135065    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:08:27.149635 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.148968    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"]
Aug 01 01:08:27.149635 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.149540    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy" containerID="cri-o://071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088" gracePeriod=30
Aug 01 01:08:27.162573 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.162046    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:08:27.201522 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.201490    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:08:27.328376 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.328348    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dc7aa1-7895-4280-93a5-3042c475cbcb-plugins-config-volume\") pod \"55dc7aa1-7895-4280-93a5-3042c475cbcb\" (UID: \"55dc7aa1-7895-4280-93a5-3042c475cbcb\") "
Aug 01 01:08:27.328570 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.328388    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d58c7\" (UniqueName: \"kubernetes.io/projected/1c9c5760-831f-48e7-9987-571d2718ef8d-kube-api-access-d58c7\") pod \"1c9c5760-831f-48e7-9987-571d2718ef8d\" (UID: \"1c9c5760-831f-48e7-9987-571d2718ef8d\") "
Aug 01 01:08:27.328570 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.328414    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls\") pod \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") "
Aug 01 01:08:27.328570 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.328456    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klxx8\" (UniqueName: \"kubernetes.io/projected/55dc7aa1-7895-4280-93a5-3042c475cbcb-kube-api-access-klxx8\") pod \"55dc7aa1-7895-4280-93a5-3042c475cbcb\" (UID: \"55dc7aa1-7895-4280-93a5-3042c475cbcb\") "
Aug 01 01:08:27.328570 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.328497    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbdzf\" (UniqueName: \"kubernetes.io/projected/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-kube-api-access-gbdzf\") pod \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") "
Aug 01 01:08:27.328803 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.328664    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9c5760-831f-48e7-9987-571d2718ef8d-plugins-config-volume\") pod \"1c9c5760-831f-48e7-9987-571d2718ef8d\" (UID: \"1c9c5760-831f-48e7-9987-571d2718ef8d\") "
Aug 01 01:08:27.328803 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.328720    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-openshift-service-ca\") pod \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\" (UID: \"4668aa5a-4cfe-4c76-b722-e06ec81d10d2\") "
Aug 01 01:08:27.328973 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.328793    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55dc7aa1-7895-4280-93a5-3042c475cbcb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55dc7aa1-7895-4280-93a5-3042c475cbcb" (UID: "55dc7aa1-7895-4280-93a5-3042c475cbcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:08:27.329089 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.329061    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9c5760-831f-48e7-9987-571d2718ef8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c9c5760-831f-48e7-9987-571d2718ef8d" (UID: "1c9c5760-831f-48e7-9987-571d2718ef8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:08:27.329159 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.329078    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dc7aa1-7895-4280-93a5-3042c475cbcb-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:27.329275 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.329234    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4668aa5a-4cfe-4c76-b722-e06ec81d10d2" (UID: "4668aa5a-4cfe-4c76-b722-e06ec81d10d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:08:27.330872 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.330850    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55dc7aa1-7895-4280-93a5-3042c475cbcb-kube-api-access-klxx8" (OuterVolumeSpecName: "kube-api-access-klxx8") pod "55dc7aa1-7895-4280-93a5-3042c475cbcb" (UID: "55dc7aa1-7895-4280-93a5-3042c475cbcb"). InnerVolumeSpecName "kube-api-access-klxx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:08:27.331005 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.330864    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4668aa5a-4cfe-4c76-b722-e06ec81d10d2" (UID: "4668aa5a-4cfe-4c76-b722-e06ec81d10d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:08:27.331005 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.330898    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-kube-api-access-gbdzf" (OuterVolumeSpecName: "kube-api-access-gbdzf") pod "4668aa5a-4cfe-4c76-b722-e06ec81d10d2" (UID: "4668aa5a-4cfe-4c76-b722-e06ec81d10d2"). InnerVolumeSpecName "kube-api-access-gbdzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:08:27.331005 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.330908    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9c5760-831f-48e7-9987-571d2718ef8d-kube-api-access-d58c7" (OuterVolumeSpecName: "kube-api-access-d58c7") pod "1c9c5760-831f-48e7-9987-571d2718ef8d" (UID: "1c9c5760-831f-48e7-9987-571d2718ef8d"). InnerVolumeSpecName "kube-api-access-d58c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:08:27.429988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.429900    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbdzf\" (UniqueName: \"kubernetes.io/projected/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-kube-api-access-gbdzf\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:27.429988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.429925    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9c5760-831f-48e7-9987-571d2718ef8d-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:27.429988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.429964    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:27.429988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.429976    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58c7\" (UniqueName: \"kubernetes.io/projected/1c9c5760-831f-48e7-9987-571d2718ef8d-kube-api-access-d58c7\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:27.430233 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.429991    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4668aa5a-4cfe-4c76-b722-e06ec81d10d2-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:27.430233 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:27.430004    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klxx8\" (UniqueName: \"kubernetes.io/projected/55dc7aa1-7895-4280-93a5-3042c475cbcb-kube-api-access-klxx8\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:28.064157 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.064127    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"
Aug 01 01:08:28.064608 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.064170    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t" event={"ID":"1c9c5760-831f-48e7-9987-571d2718ef8d","Type":"ContainerDied","Data":"ea4e8fd5a0ec72484cd16915fc312c04c2b7bb496d3973323dc9402e5755d762"}
Aug 01 01:08:28.064608 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.064231    2570 scope.go:117] "RemoveContainer" containerID="eacbb2494c2fee4c83bd0b1bdf7a7c3c5ca7a41f4d421e3728e1b68dacab8c90"
Aug 01 01:08:28.065647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.065466    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"
Aug 01 01:08:28.065647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.065482    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v" event={"ID":"4668aa5a-4cfe-4c76-b722-e06ec81d10d2","Type":"ContainerDied","Data":"4891bf69f83cc7df53106c076faaadc191fd50358a8eef71aac6a224913c1921"}
Aug 01 01:08:28.066916 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.066891    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x" event={"ID":"55dc7aa1-7895-4280-93a5-3042c475cbcb","Type":"ContainerDied","Data":"724d6bd116a3773a0bd251532c4a40e18fcb6650ffd4ea3697e57cfdf5dd958a"}
Aug 01 01:08:28.066916 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.066913    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"
Aug 01 01:08:28.074924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.074906    2570 scope.go:117] "RemoveContainer" containerID="8475bf16bc8054d0a7dae5acedf92b31154bc3c93f451615a13f52e4c1609df3"
Aug 01 01:08:28.083481 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.083461    2570 scope.go:117] "RemoveContainer" containerID="23789886248b0492d6db3fbd8a79b2a04dc14d5246e303807856f7dccc9b7a8c"
Aug 01 01:08:28.092230 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.092210    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"]
Aug 01 01:08:28.097976 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.097956    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7ed58d7f-7b667dcdb8-lsm4t"]
Aug 01 01:08:28.112261 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.112240    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"]
Aug 01 01:08:28.116487 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.116467    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7ed58d7f-9bc654c8d-xlq6x"]
Aug 01 01:08:28.129787 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.129766    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"]
Aug 01 01:08:28.136268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.136251    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7ed58d7f-688f845657-7rl9v"]
Aug 01 01:08:28.236770 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.236740    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"]
Aug 01 01:08:28.237212 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237193    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4668aa5a-4cfe-4c76-b722-e06ec81d10d2" containerName="maas-api"
Aug 01 01:08:28.237309 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237214    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4668aa5a-4cfe-4c76-b722-e06ec81d10d2" containerName="maas-api"
Aug 01 01:08:28.237309 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237231    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55dc7aa1-7895-4280-93a5-3042c475cbcb" containerName="payload-processing"
Aug 01 01:08:28.237309 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237240    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55dc7aa1-7895-4280-93a5-3042c475cbcb" containerName="payload-processing"
Aug 01 01:08:28.237309 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237253    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9c5760-831f-48e7-9987-571d2718ef8d" containerName="payload-pre-processing"
Aug 01 01:08:28.237309 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237261    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9c5760-831f-48e7-9987-571d2718ef8d" containerName="payload-pre-processing"
Aug 01 01:08:28.237576 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237376    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4668aa5a-4cfe-4c76-b722-e06ec81d10d2" containerName="maas-api"
Aug 01 01:08:28.237576 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237389    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9c5760-831f-48e7-9987-571d2718ef8d" containerName="payload-pre-processing"
Aug 01 01:08:28.237576 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.237397    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="55dc7aa1-7895-4280-93a5-3042c475cbcb" containerName="payload-processing"
Aug 01 01:08:28.240355 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.240334    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.243519 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.243495    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3fd90f77-gw-openshift-default-dockercfg-ntdsh\""
Aug 01 01:08:28.255408 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.254649    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"]
Aug 01 01:08:28.263192 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.263171    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9c5760-831f-48e7-9987-571d2718ef8d" path="/var/lib/kubelet/pods/1c9c5760-831f-48e7-9987-571d2718ef8d/volumes"
Aug 01 01:08:28.263547 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.263534    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4668aa5a-4cfe-4c76-b722-e06ec81d10d2" path="/var/lib/kubelet/pods/4668aa5a-4cfe-4c76-b722-e06ec81d10d2/volumes"
Aug 01 01:08:28.263854 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.263841    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55dc7aa1-7895-4280-93a5-3042c475cbcb" path="/var/lib/kubelet/pods/55dc7aa1-7895-4280-93a5-3042c475cbcb/volumes"
Aug 01 01:08:28.441009 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.440896    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knslp\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-kube-api-access-knslp\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.441009 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.440963    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-socket\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.441009 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.440999    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca23420-2b9c-48df-bd35-99535578df7f-istiod-ca-cert\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.441235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.441021    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-istio-token\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.441235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.441082    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-certs\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.441235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.441146    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-credential-socket\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.441235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.441169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-data\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.441235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.441189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca23420-2b9c-48df-bd35-99535578df7f-istio-podinfo\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.441400 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.441318    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-envoy\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542153 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542120    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-data\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542153 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542154    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca23420-2b9c-48df-bd35-99535578df7f-istio-podinfo\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-envoy\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542230    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knslp\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-kube-api-access-knslp\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542255    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-socket\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542273    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca23420-2b9c-48df-bd35-99535578df7f-istiod-ca-cert\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542298    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-istio-token\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542330    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-certs\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542370    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-credential-socket\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542576    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-data\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542595    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-socket\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542818 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542761    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-credential-socket\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.542818 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.542790    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-certs\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.543068 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.543044    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca23420-2b9c-48df-bd35-99535578df7f-istiod-ca-cert\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.544566 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.544546    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-envoy\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.544878 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.544861    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca23420-2b9c-48df-bd35-99535578df7f-istio-podinfo\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.549881 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.549860    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-istio-token\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.550269 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.550247    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knslp\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-kube-api-access-knslp\") pod \"e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") " pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.553018 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.552999    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:28.686365 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.686332    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"]
Aug 01 01:08:28.689393 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:08:28.689366    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca23420_2b9c_48df_bd35_99535578df7f.slice/crio-67be213e05f4e8c6bcc31b019ad384f632631f6abaab9f8c3cec75bdc6d652b5 WatchSource:0}: Error finding container 67be213e05f4e8c6bcc31b019ad384f632631f6abaab9f8c3cec75bdc6d652b5: Status 404 returned error can't find the container with id 67be213e05f4e8c6bcc31b019ad384f632631f6abaab9f8c3cec75bdc6d652b5
Aug 01 01:08:28.691436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.691373    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:08:28.691531 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.691438    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:08:28.691531 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:28.691471    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:08:29.076392 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:29.076351    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" event={"ID":"eca23420-2b9c-48df-bd35-99535578df7f","Type":"ContainerStarted","Data":"137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021"}
Aug 01 01:08:29.076837 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:29.076422    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" event={"ID":"eca23420-2b9c-48df-bd35-99535578df7f","Type":"ContainerStarted","Data":"67be213e05f4e8c6bcc31b019ad384f632631f6abaab9f8c3cec75bdc6d652b5"}
Aug 01 01:08:29.111344 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:29.111288    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" podStartSLOduration=1.111273577 podStartE2EDuration="1.111273577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:08:29.108399439 +0000 UTC m=+2075.342331705" watchObservedRunningTime="2026-08-01 01:08:29.111273577 +0000 UTC m=+2075.345205923"
Aug 01 01:08:29.554172 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:29.554083    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:29.559091 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:29.559064    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:30.082064 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.082039    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:30.083018 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.082982    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:30.663839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.663805    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"]
Aug 01 01:08:30.667955 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.667915    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:30.670356 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.670335    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3fd90f77-serving-cert\""
Aug 01 01:08:30.677024 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.677000    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"]
Aug 01 01:08:30.763592 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.763558    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:30.763801 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.763608    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbsq\" (UniqueName: \"kubernetes.io/projected/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-kube-api-access-ccbsq\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:30.763801 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.763685    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-openshift-service-ca\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:30.864991 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.864916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:30.865186 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.865032    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbsq\" (UniqueName: \"kubernetes.io/projected/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-kube-api-access-ccbsq\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:30.865186 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:08:30.865059    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-serving-cert: secret "maas-api-e2e-migrate-3fd90f77-serving-cert" not found
Aug 01 01:08:30.865186 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.865082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-openshift-service-ca\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:30.865186 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:08:30.865125    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls podName:82743b9a-b3b7-47da-b99d-8ac5427bcf7f nodeName:}" failed. No retries permitted until 2026-08-01 01:08:31.365103518 +0000 UTC m=+2077.599035778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls") pod "maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6" (UID: "82743b9a-b3b7-47da-b99d-8ac5427bcf7f") : secret "maas-api-e2e-migrate-3fd90f77-serving-cert" not found
Aug 01 01:08:30.865715 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.865694    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-openshift-service-ca\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:30.873236 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:30.873204    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbsq\" (UniqueName: \"kubernetes.io/projected/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-kube-api-access-ccbsq\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:31.312017 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.311978    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"]
Aug 01 01:08:31.315441 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.315425    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:31.318166 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.318144    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3fd90f77-dockercfg-4k8hn\""
Aug 01 01:08:31.318302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.318179    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3fd90f77\""
Aug 01 01:08:31.328462 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.328438    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"]
Aug 01 01:08:31.369108 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.369071    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:31.369299 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.369189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm\" (UID: \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:31.369299 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.369214    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npznp\" (UniqueName: \"kubernetes.io/projected/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-kube-api-access-npznp\") pod \"payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm\" (UID: \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:31.371729 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.371699    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls\") pod \"maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:31.469695 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.469657    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npznp\" (UniqueName: \"kubernetes.io/projected/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-kube-api-access-npznp\") pod \"payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm\" (UID: \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:31.469904 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.469763    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm\" (UID: \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:31.470410 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.470389    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm\" (UID: \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:31.478202 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.478177    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npznp\" (UniqueName: \"kubernetes.io/projected/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-kube-api-access-npznp\") pod \"payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm\" (UID: \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:31.580031 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.579999    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:31.626778 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.626742    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:31.725453 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.725418    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"]
Aug 01 01:08:31.729525 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:08:31.729468    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82743b9a_b3b7_47da_b99d_8ac5427bcf7f.slice/crio-531c1a95845fe988b3132219656f8c43a769b70ac5a0f7985622b634045d0cd4 WatchSource:0}: Error finding container 531c1a95845fe988b3132219656f8c43a769b70ac5a0f7985622b634045d0cd4: Status 404 returned error can't find the container with id 531c1a95845fe988b3132219656f8c43a769b70ac5a0f7985622b634045d0cd4
Aug 01 01:08:31.779085 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:31.779059    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"]
Aug 01 01:08:31.781209 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:08:31.781168    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f747cca_64ea_4c5c_92f6_76bc1144fbc1.slice/crio-e34cdb45ee9ace88882c0f5ee092973ad165e82137eb26b8f4644013dd85f30e WatchSource:0}: Error finding container e34cdb45ee9ace88882c0f5ee092973ad165e82137eb26b8f4644013dd85f30e: Status 404 returned error can't find the container with id e34cdb45ee9ace88882c0f5ee092973ad165e82137eb26b8f4644013dd85f30e
Aug 01 01:08:32.093771 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.093673    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm" event={"ID":"6f747cca-64ea-4c5c-92f6-76bc1144fbc1","Type":"ContainerStarted","Data":"3ea2c9bb88e53440a62b9d0a2960756bdff38381c5e82643ed71c6100fa041f8"}
Aug 01 01:08:32.093771 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.093748    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:32.093992 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.093772    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm" event={"ID":"6f747cca-64ea-4c5c-92f6-76bc1144fbc1","Type":"ContainerStarted","Data":"e34cdb45ee9ace88882c0f5ee092973ad165e82137eb26b8f4644013dd85f30e"}
Aug 01 01:08:32.094962 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.094913    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6" event={"ID":"82743b9a-b3b7-47da-b99d-8ac5427bcf7f","Type":"ContainerStarted","Data":"531c1a95845fe988b3132219656f8c43a769b70ac5a0f7985622b634045d0cd4"}
Aug 01 01:08:32.110228 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.110174    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm" podStartSLOduration=1.110155445 podStartE2EDuration="1.110155445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:08:32.109347131 +0000 UTC m=+2078.343279397" watchObservedRunningTime="2026-08-01 01:08:32.110155445 +0000 UTC m=+2078.344087778"
Aug 01 01:08:32.411794 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.411760    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:08:32.479539 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479511    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-envoy\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.479698 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479552    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istiod-ca-cert\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.479698 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479573    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-data\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.479786 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479758    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-certs\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.479841 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479801    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psn6c\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-kube-api-access-psn6c\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.479841 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479834    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-podinfo\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.479956 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479844    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-data" (OuterVolumeSpecName: "istio-data") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:08:32.479956 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479866    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-token\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.479956 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479912    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-credential-socket\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.480131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479969    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-socket\") pod \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\" (UID: \"67ea2abe-cb61-4f2d-848c-7c94ba7b9238\") "
Aug 01 01:08:32.480131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.479988    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:08:32.480131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.480011    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:08:32.480289 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.480179    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:08:32.480289 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.480262    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:08:32.480289 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.480285    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:32.480459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.480304    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:32.480459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.480317    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:32.480459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.480331    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:32.482036 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.482005    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:08:32.482175 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.482132    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-kube-api-access-psn6c" (OuterVolumeSpecName: "kube-api-access-psn6c") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "kube-api-access-psn6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:08:32.482299 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.482279    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-token" (OuterVolumeSpecName: "istio-token") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:08:32.482367 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.482351    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67ea2abe-cb61-4f2d-848c-7c94ba7b9238" (UID: "67ea2abe-cb61-4f2d-848c-7c94ba7b9238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:08:32.581595 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.581557    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psn6c\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-kube-api-access-psn6c\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:32.581595 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.581591    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:32.581807 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.581606    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:32.581807 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.581619    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:32.581807 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:32.581631    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ea2abe-cb61-4f2d-848c-7c94ba7b9238-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:33.101006 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.100958    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6" event={"ID":"82743b9a-b3b7-47da-b99d-8ac5427bcf7f","Type":"ContainerStarted","Data":"e9d7331861e52f20ee3807ad32fdc05d8a8257d06e7e857afc238088aa453898"}
Aug 01 01:08:33.101211 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.101047    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:33.102623 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.102600    2570 generic.go:356] "Generic (PLEG): container finished" podID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerID="071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088" exitCode=0
Aug 01 01:08:33.102735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.102641    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"
Aug 01 01:08:33.102735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.102671    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" event={"ID":"67ea2abe-cb61-4f2d-848c-7c94ba7b9238","Type":"ContainerDied","Data":"071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088"}
Aug 01 01:08:33.102735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.102709    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8" event={"ID":"67ea2abe-cb61-4f2d-848c-7c94ba7b9238","Type":"ContainerDied","Data":"aa913ddf27aa410f25ec318103e2185be778103612af570d45f960ef46bbd639"}
Aug 01 01:08:33.102890 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.102733    2570 scope.go:117] "RemoveContainer" containerID="071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088"
Aug 01 01:08:33.118539 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.118331    2570 scope.go:117] "RemoveContainer" containerID="d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841"
Aug 01 01:08:33.119514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.119467    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6" podStartSLOduration=2.725310498 podStartE2EDuration="3.119449527s" totalImagesPullingTime="394.139029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:08:33.117648065 +0000 UTC m=+2079.351580332" watchObservedRunningTime="2026-08-01 01:08:33.119449527 +0000 UTC m=+2079.353381796"
Aug 01 01:08:33.126598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.126577    2570 scope.go:117] "RemoveContainer" containerID="071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088"
Aug 01 01:08:33.126850 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:08:33.126829    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088\": container with ID starting with 071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088 not found: ID does not exist" containerID="071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088"
Aug 01 01:08:33.126992 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.126860    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088"} err="failed to get container status \"071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088\": rpc error: code = NotFound desc = could not find container \"071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088\": container with ID starting with 071349fb53de0770a1d89d63007ee1194865268a0a763fa28c5341dd9a9ba088 not found: ID does not exist"
Aug 01 01:08:33.126992 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.126883    2570 scope.go:117] "RemoveContainer" containerID="d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841"
Aug 01 01:08:33.127135 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:08:33.127117    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841\": container with ID starting with d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841 not found: ID does not exist" containerID="d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841"
Aug 01 01:08:33.127178 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.127141    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841"} err="failed to get container status \"d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841\": rpc error: code = NotFound desc = could not find container \"d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841\": container with ID starting with d23b6d5f5d020f5df32cb1dab2c78d3082f3e946bf72f5f9aee621cb2be14841 not found: ID does not exist"
Aug 01 01:08:33.132068 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.132018    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"]
Aug 01 01:08:33.137331 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:33.137310    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7ed58d7f-openshift-default-67db84f84-ffcm8"]
Aug 01 01:08:34.263175 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:34.263143    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" path="/var/lib/kubelet/pods/67ea2abe-cb61-4f2d-848c-7c94ba7b9238/volumes"
Aug 01 01:08:39.113812 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:39.113783    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:08:43.105080 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:43.105047    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:08:44.157548 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:44.157511    2570 generic.go:356] "Generic (PLEG): container finished" podID="eca23420-2b9c-48df-bd35-99535578df7f" containerID="137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021" exitCode=0
Aug 01 01:08:44.158010 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:44.157558    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" event={"ID":"eca23420-2b9c-48df-bd35-99535578df7f","Type":"ContainerDied","Data":"137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021"}
Aug 01 01:08:44.158010 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:44.157994    2570 scope.go:117] "RemoveContainer" containerID="137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021"
Aug 01 01:08:44.159102 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:44.159073    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:08:44.159180 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:44.159143    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:08:44.159220 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:44.159180    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:08:45.082686 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:45.082636    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:45.168513 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:45.168458    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" event={"ID":"eca23420-2b9c-48df-bd35-99535578df7f","Type":"ContainerStarted","Data":"237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179"}
Aug 01 01:08:45.553838 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:45.553732    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:45.555007 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:45.554975    2570 patch_prober.go:28] interesting pod/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Aug 01 01:08:45.555139 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:45.555054    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Aug 01 01:08:46.554118 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:46.554069    2570 patch_prober.go:28] interesting pod/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Aug 01 01:08:46.554638 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:46.554159    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Aug 01 01:08:47.554395 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:47.554348    2570 patch_prober.go:28] interesting pod/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Aug 01 01:08:47.554905 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:47.554432    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Aug 01 01:08:48.554078 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:48.554031    2570 patch_prober.go:28] interesting pod/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Aug 01 01:08:48.554305 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:48.554114    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Aug 01 01:08:49.554259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:49.554215    2570 patch_prober.go:28] interesting pod/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Aug 01 01:08:49.554771 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:49.554287    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Aug 01 01:08:50.558008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:50.557975    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:50.558415 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:50.558392    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:50.559040 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:50.559021    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:08:55.651975 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.651924    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"]
Aug 01 01:08:55.652507 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.652489    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy"
Aug 01 01:08:55.652567 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.652511    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy"
Aug 01 01:08:55.652567 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.652532    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy"
Aug 01 01:08:55.652567 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.652542    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy"
Aug 01 01:08:55.652664 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.652626    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy"
Aug 01 01:08:55.652664 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.652640    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ea2abe-cb61-4f2d-848c-7c94ba7b9238" containerName="istio-proxy"
Aug 01 01:08:55.656050 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.656029    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"
Aug 01 01:08:55.658599 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.658579    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\""
Aug 01 01:08:55.663545 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.663482    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"]
Aug 01 01:08:55.800228 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.800186    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b974e888-7885-4198-a833-cd670e04575f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4\" (UID: \"b974e888-7885-4198-a833-cd670e04575f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"
Aug 01 01:08:55.901157 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.901129    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b974e888-7885-4198-a833-cd670e04575f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4\" (UID: \"b974e888-7885-4198-a833-cd670e04575f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"
Aug 01 01:08:55.901739 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.901714    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b974e888-7885-4198-a833-cd670e04575f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4\" (UID: \"b974e888-7885-4198-a833-cd670e04575f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"
Aug 01 01:08:55.970189 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:55.970129    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"
Aug 01 01:08:56.099329 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:56.099298    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"]
Aug 01 01:08:56.102983 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:08:56.102949    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb974e888_7885_4198_a833_cd670e04575f.slice/crio-30356a0427dc6a0cd6742f2107f419a1d44383d14334007baa19f9ebbbad0b6e WatchSource:0}: Error finding container 30356a0427dc6a0cd6742f2107f419a1d44383d14334007baa19f9ebbbad0b6e: Status 404 returned error can't find the container with id 30356a0427dc6a0cd6742f2107f419a1d44383d14334007baa19f9ebbbad0b6e
Aug 01 01:08:56.226021 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:56.225924    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4" event={"ID":"b974e888-7885-4198-a833-cd670e04575f","Type":"ContainerStarted","Data":"30356a0427dc6a0cd6742f2107f419a1d44383d14334007baa19f9ebbbad0b6e"}
Aug 01 01:08:57.232446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:57.232409    2570 generic.go:356] "Generic (PLEG): container finished" podID="b974e888-7885-4198-a833-cd670e04575f" containerID="b970ec45ba0f1d72ec4d999f4bd8222f02a75f591b8b6e1a348a88d96c322b2e" exitCode=0
Aug 01 01:08:57.232859 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:57.232490    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4" event={"ID":"b974e888-7885-4198-a833-cd670e04575f","Type":"ContainerDied","Data":"b970ec45ba0f1d72ec4d999f4bd8222f02a75f591b8b6e1a348a88d96c322b2e"}
Aug 01 01:08:58.367461 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:58.367432    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"
Aug 01 01:08:58.523323 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:58.523233    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b974e888-7885-4198-a833-cd670e04575f-maas-api-service-ca\") pod \"b974e888-7885-4198-a833-cd670e04575f\" (UID: \"b974e888-7885-4198-a833-cd670e04575f\") "
Aug 01 01:08:58.523637 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:58.523610    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b974e888-7885-4198-a833-cd670e04575f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b974e888-7885-4198-a833-cd670e04575f" (UID: "b974e888-7885-4198-a833-cd670e04575f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:08:58.624420 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:58.624382    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b974e888-7885-4198-a833-cd670e04575f-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:08:59.242362 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:59.242331    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"
Aug 01 01:08:59.242516 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:59.242361    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4" event={"ID":"b974e888-7885-4198-a833-cd670e04575f","Type":"ContainerDied","Data":"30356a0427dc6a0cd6742f2107f419a1d44383d14334007baa19f9ebbbad0b6e"}
Aug 01 01:08:59.242516 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:08:59.242429    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30356a0427dc6a0cd6742f2107f419a1d44383d14334007baa19f9ebbbad0b6e"
Aug 01 01:09:01.990136 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:01.990102    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"]
Aug 01 01:09:01.990745 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:01.990326    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6" podUID="82743b9a-b3b7-47da-b99d-8ac5427bcf7f" containerName="maas-api" containerID="cri-o://e9d7331861e52f20ee3807ad32fdc05d8a8257d06e7e857afc238088aa453898" gracePeriod=30
Aug 01 01:09:02.054360 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.054313    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"]
Aug 01 01:09:02.054827 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.054557    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm" podUID="6f747cca-64ea-4c5c-92f6-76bc1144fbc1" containerName="payload-pre-processing" containerID="cri-o://3ea2c9bb88e53440a62b9d0a2960756bdff38381c5e82643ed71c6100fa041f8" gracePeriod=30
Aug 01 01:09:02.184744 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.184702    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"]
Aug 01 01:09:02.188301 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.188271    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3fd90f77-hp2n4"]
Aug 01 01:09:02.263322 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.263083    2570 generic.go:356] "Generic (PLEG): container finished" podID="82743b9a-b3b7-47da-b99d-8ac5427bcf7f" containerID="e9d7331861e52f20ee3807ad32fdc05d8a8257d06e7e857afc238088aa453898" exitCode=0
Aug 01 01:09:02.265488 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.265454    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b974e888-7885-4198-a833-cd670e04575f" path="/var/lib/kubelet/pods/b974e888-7885-4198-a833-cd670e04575f/volumes"
Aug 01 01:09:02.266079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.266050    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6" event={"ID":"82743b9a-b3b7-47da-b99d-8ac5427bcf7f","Type":"ContainerDied","Data":"e9d7331861e52f20ee3807ad32fdc05d8a8257d06e7e857afc238088aa453898"}
Aug 01 01:09:02.267031 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.266998    2570 generic.go:356] "Generic (PLEG): container finished" podID="6f747cca-64ea-4c5c-92f6-76bc1144fbc1" containerID="3ea2c9bb88e53440a62b9d0a2960756bdff38381c5e82643ed71c6100fa041f8" exitCode=0
Aug 01 01:09:02.267281 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.267048    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm" event={"ID":"6f747cca-64ea-4c5c-92f6-76bc1144fbc1","Type":"ContainerDied","Data":"3ea2c9bb88e53440a62b9d0a2960756bdff38381c5e82643ed71c6100fa041f8"}
Aug 01 01:09:02.327069 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.327034    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"]
Aug 01 01:09:02.327328 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.327293    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy" containerID="cri-o://237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179" gracePeriod=30
Aug 01 01:09:02.335102 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.335064    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:09:02.345265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.345235    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:09:02.455806 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.455773    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-plugins-config-volume\") pod \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\" (UID: \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\") "
Aug 01 01:09:02.456008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.455827    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npznp\" (UniqueName: \"kubernetes.io/projected/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-kube-api-access-npznp\") pod \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\" (UID: \"6f747cca-64ea-4c5c-92f6-76bc1144fbc1\") "
Aug 01 01:09:02.456008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.455882    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccbsq\" (UniqueName: \"kubernetes.io/projected/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-kube-api-access-ccbsq\") pod \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") "
Aug 01 01:09:02.456008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.455905    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-openshift-service-ca\") pod \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") "
Aug 01 01:09:02.456008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.455957    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls\") pod \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\" (UID: \"82743b9a-b3b7-47da-b99d-8ac5427bcf7f\") "
Aug 01 01:09:02.456250 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.456201    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f747cca-64ea-4c5c-92f6-76bc1144fbc1" (UID: "6f747cca-64ea-4c5c-92f6-76bc1144fbc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:09:02.456309 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.456275    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82743b9a-b3b7-47da-b99d-8ac5427bcf7f" (UID: "82743b9a-b3b7-47da-b99d-8ac5427bcf7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:09:02.458027 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.458000    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-kube-api-access-ccbsq" (OuterVolumeSpecName: "kube-api-access-ccbsq") pod "82743b9a-b3b7-47da-b99d-8ac5427bcf7f" (UID: "82743b9a-b3b7-47da-b99d-8ac5427bcf7f"). InnerVolumeSpecName "kube-api-access-ccbsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:09:02.458027 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.458012    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-kube-api-access-npznp" (OuterVolumeSpecName: "kube-api-access-npznp") pod "6f747cca-64ea-4c5c-92f6-76bc1144fbc1" (UID: "6f747cca-64ea-4c5c-92f6-76bc1144fbc1"). InnerVolumeSpecName "kube-api-access-npznp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:09:02.458178 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.458032    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82743b9a-b3b7-47da-b99d-8ac5427bcf7f" (UID: "82743b9a-b3b7-47da-b99d-8ac5427bcf7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:09:02.557514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.557428    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:02.557514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.557458    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npznp\" (UniqueName: \"kubernetes.io/projected/6f747cca-64ea-4c5c-92f6-76bc1144fbc1-kube-api-access-npznp\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:02.557514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.557470    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccbsq\" (UniqueName: \"kubernetes.io/projected/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-kube-api-access-ccbsq\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:02.557514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.557480    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:02.557514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:02.557489    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82743b9a-b3b7-47da-b99d-8ac5427bcf7f-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:03.121078 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121042    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"]
Aug 01 01:09:03.121433 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121399    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f747cca-64ea-4c5c-92f6-76bc1144fbc1" containerName="payload-pre-processing"
Aug 01 01:09:03.121433 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121411    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f747cca-64ea-4c5c-92f6-76bc1144fbc1" containerName="payload-pre-processing"
Aug 01 01:09:03.121433 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121424    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82743b9a-b3b7-47da-b99d-8ac5427bcf7f" containerName="maas-api"
Aug 01 01:09:03.121433 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121429    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="82743b9a-b3b7-47da-b99d-8ac5427bcf7f" containerName="maas-api"
Aug 01 01:09:03.121560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121439    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b974e888-7885-4198-a833-cd670e04575f" containerName="revoke-keys"
Aug 01 01:09:03.121560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121445    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b974e888-7885-4198-a833-cd670e04575f" containerName="revoke-keys"
Aug 01 01:09:03.121560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121517    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f747cca-64ea-4c5c-92f6-76bc1144fbc1" containerName="payload-pre-processing"
Aug 01 01:09:03.121560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121528    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="82743b9a-b3b7-47da-b99d-8ac5427bcf7f" containerName="maas-api"
Aug 01 01:09:03.121560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.121535    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b974e888-7885-4198-a833-cd670e04575f" containerName="revoke-keys"
Aug 01 01:09:03.125094 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.125073    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.128077 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.128058    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8e664d6c-openshift-default-dockercfg-ngzhg\""
Aug 01 01:09:03.135964 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.135942    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"]
Aug 01 01:09:03.263797 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.263707    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-istio-token\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.263797 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.263746    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-envoy\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.263797 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.263770    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-certs\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.264076 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.263817    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-socket\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.264076 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.263844    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxsfr\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-kube-api-access-bxsfr\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.264076 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.263863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-data\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.264076 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.263912    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-credential-socket\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.264076 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.263970    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7300acfe-7970-4061-a187-239a7107d83f-istiod-ca-cert\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.264076 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.264012    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7300acfe-7970-4061-a187-239a7107d83f-istio-podinfo\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.274312 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.274250    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"
Aug 01 01:09:03.274312 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.274270    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6" event={"ID":"82743b9a-b3b7-47da-b99d-8ac5427bcf7f","Type":"ContainerDied","Data":"531c1a95845fe988b3132219656f8c43a769b70ac5a0f7985622b634045d0cd4"}
Aug 01 01:09:03.274760 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.274334    2570 scope.go:117] "RemoveContainer" containerID="e9d7331861e52f20ee3807ad32fdc05d8a8257d06e7e857afc238088aa453898"
Aug 01 01:09:03.275848 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.275826    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm" event={"ID":"6f747cca-64ea-4c5c-92f6-76bc1144fbc1","Type":"ContainerDied","Data":"e34cdb45ee9ace88882c0f5ee092973ad165e82137eb26b8f4644013dd85f30e"}
Aug 01 01:09:03.275923 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.275884    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"
Aug 01 01:09:03.284868 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.284850    2570 scope.go:117] "RemoveContainer" containerID="3ea2c9bb88e53440a62b9d0a2960756bdff38381c5e82643ed71c6100fa041f8"
Aug 01 01:09:03.299565 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.299545    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"]
Aug 01 01:09:03.303270 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.303248    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3fd90f77-589c8d9f88-d4cm6"]
Aug 01 01:09:03.316563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.316544    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"]
Aug 01 01:09:03.319099 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.319079    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3fd90f77-5fb4746c6-nvgbm"]
Aug 01 01:09:03.365069 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365034    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7300acfe-7970-4061-a187-239a7107d83f-istio-podinfo\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-istio-token\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365220    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-envoy\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365260    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-certs\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365449 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365292    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-socket\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365449 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365330    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxsfr\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-kube-api-access-bxsfr\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365449 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365375    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-data\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365449 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365407    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-credential-socket\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365449 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365433    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7300acfe-7970-4061-a187-239a7107d83f-istiod-ca-cert\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365715 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365700    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-socket\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.365866 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.365843    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-data\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.366138 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.366115    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-certs\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.366238 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.366120    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-credential-socket\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.366238 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.366192    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7300acfe-7970-4061-a187-239a7107d83f-istiod-ca-cert\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.367577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.367554    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-envoy\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.367707 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.367691    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7300acfe-7970-4061-a187-239a7107d83f-istio-podinfo\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.373086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.373063    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-istio-token\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.373189 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.373172    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxsfr\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-kube-api-access-bxsfr\") pod \"e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") " pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.436168 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.436132    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:03.571338 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.571310    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"]
Aug 01 01:09:03.573406 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:09:03.573383    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7300acfe_7970_4061_a187_239a7107d83f.slice/crio-1e7e96ce0cf56e51ad4f80a665e4a5f5411038d81184c0e7d97fd5de101547ad WatchSource:0}: Error finding container 1e7e96ce0cf56e51ad4f80a665e4a5f5411038d81184c0e7d97fd5de101547ad: Status 404 returned error can't find the container with id 1e7e96ce0cf56e51ad4f80a665e4a5f5411038d81184c0e7d97fd5de101547ad
Aug 01 01:09:03.575558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.575527    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:03.575646 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.575594    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:03.575646 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:03.575625    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:04.263895 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:04.263862    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f747cca-64ea-4c5c-92f6-76bc1144fbc1" path="/var/lib/kubelet/pods/6f747cca-64ea-4c5c-92f6-76bc1144fbc1/volumes"
Aug 01 01:09:04.264291 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:04.264274    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82743b9a-b3b7-47da-b99d-8ac5427bcf7f" path="/var/lib/kubelet/pods/82743b9a-b3b7-47da-b99d-8ac5427bcf7f/volumes"
Aug 01 01:09:04.282742 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:04.282709    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" event={"ID":"7300acfe-7970-4061-a187-239a7107d83f","Type":"ContainerStarted","Data":"11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb"}
Aug 01 01:09:04.282880 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:04.282764    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" event={"ID":"7300acfe-7970-4061-a187-239a7107d83f","Type":"ContainerStarted","Data":"1e7e96ce0cf56e51ad4f80a665e4a5f5411038d81184c0e7d97fd5de101547ad"}
Aug 01 01:09:04.303940 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:04.303889    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podStartSLOduration=1.303875838 podStartE2EDuration="1.303875838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:04.30172728 +0000 UTC m=+2110.535659546" watchObservedRunningTime="2026-08-01 01:09:04.303875838 +0000 UTC m=+2110.537808180"
Aug 01 01:09:04.437133 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:04.437098    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:04.441841 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:04.441818    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:05.289993 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:05.289949    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:05.291042 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:05.291020    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:05.560726 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:05.560649    2570 patch_prober.go:28] interesting pod/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Aug 01 01:09:05.560726 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:05.560705    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 01:09:06.723332 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.723294    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"]
Aug 01 01:09:06.728361 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.728340    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:06.730688 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.730664    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8e664d6c-serving-cert\""
Aug 01 01:09:06.736669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.736290    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"]
Aug 01 01:09:06.894396 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.894359    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-openshift-service-ca\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:06.894557 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.894434    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:06.894557 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.894467    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62w2\" (UniqueName: \"kubernetes.io/projected/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-kube-api-access-c62w2\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:06.996045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.995956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62w2\" (UniqueName: \"kubernetes.io/projected/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-kube-api-access-c62w2\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:06.996045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.996034    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-openshift-service-ca\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:06.996275 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.996080    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:06.996275 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:06.996173    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-serving-cert: secret "maas-api-e2e-ait-8e664d6c-serving-cert" not found
Aug 01 01:09:06.996275 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:06.996228    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls podName:d9d66aa3-c0ba-4cd4-a068-fb1db870b351 nodeName:}" failed. No retries permitted until 2026-08-01 01:09:07.496210541 +0000 UTC m=+2113.730142785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls") pod "maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4" (UID: "d9d66aa3-c0ba-4cd4-a068-fb1db870b351") : secret "maas-api-e2e-ait-8e664d6c-serving-cert" not found
Aug 01 01:09:06.996702 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:06.996681    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-openshift-service-ca\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:07.004598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.004571    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62w2\" (UniqueName: \"kubernetes.io/projected/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-kube-api-access-c62w2\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:07.488995 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.488965    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"]
Aug 01 01:09:07.492358 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.492338    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:07.495102 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.495075    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8e664d6c-dockercfg-h7pf7\""
Aug 01 01:09:07.495303 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.495282    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8e664d6c\""
Aug 01 01:09:07.500378 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.500290    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:07.504417 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.503734    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"]
Aug 01 01:09:07.504573 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.504555    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls\") pod \"maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:07.601288 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.601256    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f\" (UID: \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:07.601431 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.601306    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-kube-api-access-5kq7x\") pod \"payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f\" (UID: \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:07.616719 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.616696    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:09:07.641211 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.641153    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:07.702565 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.702414    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f\" (UID: \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:07.702565 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.702467    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-kube-api-access-5kq7x\") pod \"payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f\" (UID: \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:07.703504 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.703352    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f\" (UID: \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:07.714153 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.714127    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-kube-api-access-5kq7x\") pod \"payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f\" (UID: \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:07.781066 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.781041    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"]
Aug 01 01:09:07.783025 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:09:07.782989    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d66aa3_c0ba_4cd4_a068_fb1db870b351.slice/crio-429861513b63d72d08afe5c6a6e39adba5f11d044aac8798d3134f46cb43d3c0 WatchSource:0}: Error finding container 429861513b63d72d08afe5c6a6e39adba5f11d044aac8798d3134f46cb43d3c0: Status 404 returned error can't find the container with id 429861513b63d72d08afe5c6a6e39adba5f11d044aac8798d3134f46cb43d3c0
Aug 01 01:09:07.803098 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803076    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca23420-2b9c-48df-bd35-99535578df7f-istiod-ca-cert\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803218 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803127    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knslp\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-kube-api-access-knslp\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803218 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803147    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-certs\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803218 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803171    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-socket\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803218 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803194    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-istio-token\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803218 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803209    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-data\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803474 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803239    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca23420-2b9c-48df-bd35-99535578df7f-istio-podinfo\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803474 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803270    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-envoy\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803474 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803301    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-credential-socket\") pod \"eca23420-2b9c-48df-bd35-99535578df7f\" (UID: \"eca23420-2b9c-48df-bd35-99535578df7f\") "
Aug 01 01:09:07.803626 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803577    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:07.803626 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803588    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca23420-2b9c-48df-bd35-99535578df7f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:09:07.803626 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803601    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-data" (OuterVolumeSpecName: "istio-data") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:07.803764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803661    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:07.803907 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.803886    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:07.805037 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.805002    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:07.805579 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.805545    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eca23420-2b9c-48df-bd35-99535578df7f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:09:07.805684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.805638    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:07.805737 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.805720    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-kube-api-access-knslp" (OuterVolumeSpecName: "kube-api-access-knslp") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "kube-api-access-knslp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:09:07.805905 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.805878    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-istio-token" (OuterVolumeSpecName: "istio-token") pod "eca23420-2b9c-48df-bd35-99535578df7f" (UID: "eca23420-2b9c-48df-bd35-99535578df7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:09:07.904232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904125    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knslp\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-kube-api-access-knslp\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.904232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904162    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.904232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904180    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.904232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904194    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca23420-2b9c-48df-bd35-99535578df7f-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.904232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904209    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.904232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904223    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca23420-2b9c-48df-bd35-99535578df7f-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.904232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904235    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.904738 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904250    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca23420-2b9c-48df-bd35-99535578df7f-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.904738 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.904264    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca23420-2b9c-48df-bd35-99535578df7f-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:07.938362 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:07.938327    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"]
Aug 01 01:09:07.940158 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:09:07.940131    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ada5cc6_fbcf_4122_ba5b_c74161c6d6c3.slice/crio-634f31e2cd6a53139a94107e9f9204cf07c7d9730aedb02f59211ced6344241c WatchSource:0}: Error finding container 634f31e2cd6a53139a94107e9f9204cf07c7d9730aedb02f59211ced6344241c: Status 404 returned error can't find the container with id 634f31e2cd6a53139a94107e9f9204cf07c7d9730aedb02f59211ced6344241c
Aug 01 01:09:08.304669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.304632    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4" event={"ID":"d9d66aa3-c0ba-4cd4-a068-fb1db870b351","Type":"ContainerStarted","Data":"fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7"}
Aug 01 01:09:08.304825 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.304706    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:08.304825 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.304727    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4" event={"ID":"d9d66aa3-c0ba-4cd4-a068-fb1db870b351","Type":"ContainerStarted","Data":"429861513b63d72d08afe5c6a6e39adba5f11d044aac8798d3134f46cb43d3c0"}
Aug 01 01:09:08.306273 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.306250    2570 generic.go:356] "Generic (PLEG): container finished" podID="eca23420-2b9c-48df-bd35-99535578df7f" containerID="237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179" exitCode=0
Aug 01 01:09:08.306376 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.306303    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"
Aug 01 01:09:08.306376 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.306319    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" event={"ID":"eca23420-2b9c-48df-bd35-99535578df7f","Type":"ContainerDied","Data":"237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179"}
Aug 01 01:09:08.306376 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.306361    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x" event={"ID":"eca23420-2b9c-48df-bd35-99535578df7f","Type":"ContainerDied","Data":"67be213e05f4e8c6bcc31b019ad384f632631f6abaab9f8c3cec75bdc6d652b5"}
Aug 01 01:09:08.306617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.306392    2570 scope.go:117] "RemoveContainer" containerID="237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179"
Aug 01 01:09:08.308081 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.308057    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f" event={"ID":"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3","Type":"ContainerStarted","Data":"926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc"}
Aug 01 01:09:08.308263 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.308233    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:08.308378 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.308274    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f" event={"ID":"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3","Type":"ContainerStarted","Data":"634f31e2cd6a53139a94107e9f9204cf07c7d9730aedb02f59211ced6344241c"}
Aug 01 01:09:08.315562 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.315539    2570 scope.go:117] "RemoveContainer" containerID="137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021"
Aug 01 01:09:08.322353 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.322305    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4" podStartSLOduration=1.891761274 podStartE2EDuration="2.322290914s" totalImagesPullingTime="430.52964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:08.320301205 +0000 UTC m=+2114.554233472" watchObservedRunningTime="2026-08-01 01:09:08.322290914 +0000 UTC m=+2114.556223180"
Aug 01 01:09:08.323918 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.323903    2570 scope.go:117] "RemoveContainer" containerID="237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179"
Aug 01 01:09:08.324221 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:08.324200    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179\": container with ID starting with 237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179 not found: ID does not exist" containerID="237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179"
Aug 01 01:09:08.324265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.324231    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179"} err="failed to get container status \"237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179\": rpc error: code = NotFound desc = could not find container \"237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179\": container with ID starting with 237777e79c2c9b945b1a7759e348dedc48caf387c44b44de316d89a6a2ff2179 not found: ID does not exist"
Aug 01 01:09:08.324265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.324250    2570 scope.go:117] "RemoveContainer" containerID="137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021"
Aug 01 01:09:08.324488 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:08.324471    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021\": container with ID starting with 137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021 not found: ID does not exist" containerID="137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021"
Aug 01 01:09:08.324532 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.324494    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021"} err="failed to get container status \"137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021\": rpc error: code = NotFound desc = could not find container \"137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021\": container with ID starting with 137180abdcd6c26c027749e91cd6ca1f79ce6402123b2cf6ea9efef9b87f5021 not found: ID does not exist"
Aug 01 01:09:08.335772 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.335731    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f" podStartSLOduration=1.335718379 podStartE2EDuration="1.335718379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:08.335651491 +0000 UTC m=+2114.569583758" watchObservedRunningTime="2026-08-01 01:09:08.335718379 +0000 UTC m=+2114.569650644"
Aug 01 01:09:08.348961 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.348924    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"]
Aug 01 01:09:08.352691 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:08.352671    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3fd90f77-gw-openshift-default-7769454756-vz28x"]
Aug 01 01:09:10.262505 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:10.262466    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eca23420-2b9c-48df-bd35-99535578df7f" path="/var/lib/kubelet/pods/eca23420-2b9c-48df-bd35-99535578df7f/volumes"
Aug 01 01:09:14.319564 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:14.319534    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:19.315389 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:19.315361    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:19.367286 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:19.367253    2570 generic.go:356] "Generic (PLEG): container finished" podID="7300acfe-7970-4061-a187-239a7107d83f" containerID="11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb" exitCode=0
Aug 01 01:09:19.367449 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:19.367311    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" event={"ID":"7300acfe-7970-4061-a187-239a7107d83f","Type":"ContainerDied","Data":"11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb"}
Aug 01 01:09:19.367716 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:19.367698    2570 scope.go:117] "RemoveContainer" containerID="11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb"
Aug 01 01:09:19.368683 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:19.368653    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:19.368757 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:19.368740    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:19.368808 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:19.368791    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:20.290119 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:20.290082    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:20.373638 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:20.373600    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" event={"ID":"7300acfe-7970-4061-a187-239a7107d83f","Type":"ContainerStarted","Data":"ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688"}
Aug 01 01:09:20.437123 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:20.437085    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:20.438315 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:20.438290    2570 patch_prober.go:28] interesting pod/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Aug 01 01:09:20.438387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:20.438342    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Aug 01 01:09:21.436819 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:21.436778    2570 patch_prober.go:28] interesting pod/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Aug 01 01:09:21.437316 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:21.436835    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Aug 01 01:09:22.436751 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:22.436718    2570 patch_prober.go:28] interesting pod/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Aug 01 01:09:22.436917 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:22.436773    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Aug 01 01:09:23.437477 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:23.437440    2570 patch_prober.go:28] interesting pod/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Aug 01 01:09:23.437885 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:23.437495    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Aug 01 01:09:24.436494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:24.436460    2570 patch_prober.go:28] interesting pod/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Aug 01 01:09:24.436666 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:24.436516    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Aug 01 01:09:25.437238 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:25.437193    2570 patch_prober.go:28] interesting pod/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Aug 01 01:09:25.437685 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:25.437255    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Aug 01 01:09:26.441445 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:26.441412    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:26.441925 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:26.441673    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:26.442495 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:26.442476    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:37.383205 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.383170    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"]
Aug 01 01:09:37.383839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.383383    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4" podUID="d9d66aa3-c0ba-4cd4-a068-fb1db870b351" containerName="maas-api" containerID="cri-o://fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7" gracePeriod=30
Aug 01 01:09:37.442792 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.442757    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"]
Aug 01 01:09:37.443031 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.443000    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f" podUID="0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3" containerName="payload-processing" containerID="cri-o://926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc" gracePeriod=30
Aug 01 01:09:37.701984 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.701956    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:37.720183 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.720153    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:37.789246 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.789209    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-kube-api-access-5kq7x\") pod \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\" (UID: \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\") "
Aug 01 01:09:37.789431 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.789268    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-openshift-service-ca\") pod \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") "
Aug 01 01:09:37.789431 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.789317    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls\") pod \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") "
Aug 01 01:09:37.789431 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.789347    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c62w2\" (UniqueName: \"kubernetes.io/projected/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-kube-api-access-c62w2\") pod \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\" (UID: \"d9d66aa3-c0ba-4cd4-a068-fb1db870b351\") "
Aug 01 01:09:37.789431 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.789377    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-plugins-config-volume\") pod \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\" (UID: \"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3\") "
Aug 01 01:09:37.789659 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.789616    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9d66aa3-c0ba-4cd4-a068-fb1db870b351" (UID: "d9d66aa3-c0ba-4cd4-a068-fb1db870b351"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:09:37.789904 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.789865    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3" (UID: "0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:09:37.791550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.791518    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-kube-api-access-5kq7x" (OuterVolumeSpecName: "kube-api-access-5kq7x") pod "0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3" (UID: "0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3"). InnerVolumeSpecName "kube-api-access-5kq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:09:37.791550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.791527    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9d66aa3-c0ba-4cd4-a068-fb1db870b351" (UID: "d9d66aa3-c0ba-4cd4-a068-fb1db870b351"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:09:37.791683 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.791548    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-kube-api-access-c62w2" (OuterVolumeSpecName: "kube-api-access-c62w2") pod "d9d66aa3-c0ba-4cd4-a068-fb1db870b351" (UID: "d9d66aa3-c0ba-4cd4-a068-fb1db870b351"). InnerVolumeSpecName "kube-api-access-c62w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:09:37.890532 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.890490    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c62w2\" (UniqueName: \"kubernetes.io/projected/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-kube-api-access-c62w2\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:37.890532 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.890524    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:37.890532 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.890533    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kq7x\" (UniqueName: \"kubernetes.io/projected/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3-kube-api-access-5kq7x\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:37.890756 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.890543    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:37.890756 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:37.890553    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d66aa3-c0ba-4cd4-a068-fb1db870b351-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:38.463885 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.463848    2570 generic.go:356] "Generic (PLEG): container finished" podID="d9d66aa3-c0ba-4cd4-a068-fb1db870b351" containerID="fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7" exitCode=0
Aug 01 01:09:38.464363 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.463893    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"
Aug 01 01:09:38.464363 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.463945    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4" event={"ID":"d9d66aa3-c0ba-4cd4-a068-fb1db870b351","Type":"ContainerDied","Data":"fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7"}
Aug 01 01:09:38.464363 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.464007    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4" event={"ID":"d9d66aa3-c0ba-4cd4-a068-fb1db870b351","Type":"ContainerDied","Data":"429861513b63d72d08afe5c6a6e39adba5f11d044aac8798d3134f46cb43d3c0"}
Aug 01 01:09:38.464363 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.464031    2570 scope.go:117] "RemoveContainer" containerID="fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7"
Aug 01 01:09:38.465459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.465273    2570 generic.go:356] "Generic (PLEG): container finished" podID="0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3" containerID="926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc" exitCode=0
Aug 01 01:09:38.465459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.465314    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"
Aug 01 01:09:38.465459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.465325    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f" event={"ID":"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3","Type":"ContainerDied","Data":"926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc"}
Aug 01 01:09:38.465459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.465362    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f" event={"ID":"0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3","Type":"ContainerDied","Data":"634f31e2cd6a53139a94107e9f9204cf07c7d9730aedb02f59211ced6344241c"}
Aug 01 01:09:38.473114 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.473087    2570 scope.go:117] "RemoveContainer" containerID="fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7"
Aug 01 01:09:38.473370 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:38.473346    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7\": container with ID starting with fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7 not found: ID does not exist" containerID="fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7"
Aug 01 01:09:38.473460 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.473378    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7"} err="failed to get container status \"fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7\": rpc error: code = NotFound desc = could not find container \"fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7\": container with ID starting with fa3204ce873a4cb119b976f8f6939ec8e3d35ab0c8a27fd1f824a38f281a51e7 not found: ID does not exist"
Aug 01 01:09:38.473460 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.473395    2570 scope.go:117] "RemoveContainer" containerID="926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc"
Aug 01 01:09:38.481042 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.481024    2570 scope.go:117] "RemoveContainer" containerID="926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc"
Aug 01 01:09:38.481274 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:38.481255    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc\": container with ID starting with 926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc not found: ID does not exist" containerID="926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc"
Aug 01 01:09:38.481327 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.481281    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc"} err="failed to get container status \"926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc\": rpc error: code = NotFound desc = could not find container \"926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc\": container with ID starting with 926aca528704235f54f6ef9299747a9f5bf07c51f794bb324755af78aefbcbbc not found: ID does not exist"
Aug 01 01:09:38.487805 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.487777    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"]
Aug 01 01:09:38.492002 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.491979    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e664d6c-5cf85ccbf9-656l4"]
Aug 01 01:09:38.506951 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.506905    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"]
Aug 01 01:09:38.514747 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.514719    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e664d6c-7696b76984-mrv9f"]
Aug 01 01:09:38.865276 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.865238    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"]
Aug 01 01:09:38.865534 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:38.865505    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy" containerID="cri-o://ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688" gracePeriod=30
Aug 01 01:09:39.661188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661149    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"]
Aug 01 01:09:39.661764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661696    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3" containerName="payload-processing"
Aug 01 01:09:39.661764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661713    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3" containerName="payload-processing"
Aug 01 01:09:39.661764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661731    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy"
Aug 01 01:09:39.661764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661741    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy"
Aug 01 01:09:39.661764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661764    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy"
Aug 01 01:09:39.662082 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661772    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy"
Aug 01 01:09:39.662082 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661787    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d66aa3-c0ba-4cd4-a068-fb1db870b351" containerName="maas-api"
Aug 01 01:09:39.662082 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661795    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d66aa3-c0ba-4cd4-a068-fb1db870b351" containerName="maas-api"
Aug 01 01:09:39.662082 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661876    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d66aa3-c0ba-4cd4-a068-fb1db870b351" containerName="maas-api"
Aug 01 01:09:39.662082 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661887    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy"
Aug 01 01:09:39.662082 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661899    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca23420-2b9c-48df-bd35-99535578df7f" containerName="istio-proxy"
Aug 01 01:09:39.662082 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.661910    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3" containerName="payload-processing"
Aug 01 01:09:39.665178 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.665151    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.668088 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.668068    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ef000e4f-openshift-default-dockercfg-g9p9r\""
Aug 01 01:09:39.677139 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.677111    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"]
Aug 01 01:09:39.810033 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.809980    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-certs\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.810265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.810056    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-socket\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.810265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.810100    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-podinfo\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.810265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.810164    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-credential-socket\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.810265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.810190    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-token\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.810265 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.810235    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-envoy\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.810503 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.810323    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69sdt\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-kube-api-access-69sdt\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.810503 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.810363    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istiod-ca-cert\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.810503 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.810387    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-data\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.911601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911505    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-certs\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.911601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-socket\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.911787 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911616    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-podinfo\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.911787 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911660    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-credential-socket\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.911787 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911683    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-token\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.911787 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911730    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-envoy\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.911787 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911784    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69sdt\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-kube-api-access-69sdt\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.912086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911810    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istiod-ca-cert\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.912086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.911840    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-data\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.912086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.912042    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-socket\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.912086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.912072    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-certs\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.912302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.912118    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-credential-socket\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.912302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.912214    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-data\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.912636 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.912608    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istiod-ca-cert\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.914250 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.914218    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-podinfo\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.914342 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.914293    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-envoy\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.920617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.920592    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-token\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.921160 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.921130    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69sdt\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-kube-api-access-69sdt\") pod \"e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") " pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:39.980225 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:39.980183    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:40.120459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.120429    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"]
Aug 01 01:09:40.126811 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.126772    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:40.126903 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.126845    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:40.126903 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.126880    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:40.262469 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.262432    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3" path="/var/lib/kubelet/pods/0ada5cc6-fbcf-4122-ba5b-c74161c6d6c3/volumes"
Aug 01 01:09:40.262870 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.262855    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d66aa3-c0ba-4cd4-a068-fb1db870b351" path="/var/lib/kubelet/pods/d9d66aa3-c0ba-4cd4-a068-fb1db870b351/volumes"
Aug 01 01:09:40.477998 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.477904    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" event={"ID":"dc8efb4b-9be1-4b69-9a7e-ea842093deb4","Type":"ContainerStarted","Data":"d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d"}
Aug 01 01:09:40.477998 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.477977    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" event={"ID":"dc8efb4b-9be1-4b69-9a7e-ea842093deb4","Type":"ContainerStarted","Data":"6458558330af52e07feb9520f3dfe262ea9cbdc44d1862987611d0fb8b06a772"}
Aug 01 01:09:40.500178 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.500125    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" podStartSLOduration=1.5001092040000001 podStartE2EDuration="1.500109204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:40.499655301 +0000 UTC m=+2146.733587567" watchObservedRunningTime="2026-08-01 01:09:40.500109204 +0000 UTC m=+2146.734041470"
Aug 01 01:09:40.981015 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:40.980976    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:41.444097 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:41.444058    2570 patch_prober.go:28] interesting pod/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Aug 01 01:09:41.444282 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:41.444128    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 01:09:41.985403 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:41.985374    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:42.488003 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:42.487965    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:42.489100 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:42.489082    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:42.970223 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:42.970184    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"]
Aug 01 01:09:42.974696 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:42.974670    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:42.977259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:42.977238    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ef000e4f-serving-cert\""
Aug 01 01:09:42.982587 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:42.982556    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"]
Aug 01 01:09:43.043227 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.043190    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-openshift-service-ca\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.043227 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.043241    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgqg9\" (UniqueName: \"kubernetes.io/projected/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-kube-api-access-dgqg9\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.043684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.043306    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.144610 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.144571    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.144808 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.144690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-openshift-service-ca\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.144808 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:43.144737    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-serving-cert: secret "maas-api-e2e-derive-ef000e4f-serving-cert" not found
Aug 01 01:09:43.144808 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.144736    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgqg9\" (UniqueName: \"kubernetes.io/projected/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-kube-api-access-dgqg9\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.144808 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:43.144808    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls podName:6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7 nodeName:}" failed. No retries permitted until 2026-08-01 01:09:43.644786624 +0000 UTC m=+2149.878718886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls") pod "maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x" (UID: "6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7") : secret "maas-api-e2e-derive-ef000e4f-serving-cert" not found
Aug 01 01:09:43.145393 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.145371    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-openshift-service-ca\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.156831 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.156802    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgqg9\" (UniqueName: \"kubernetes.io/projected/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-kube-api-access-dgqg9\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.650470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.650429    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.652909 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.652882    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls\") pod \"maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:43.890093 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:43.890065    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:44.032490 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.032460    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"]
Aug 01 01:09:44.034379 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:09:44.034352    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e09d91f_dcf3_41bf_8f93_3bc6dab0cff7.slice/crio-b31ffc499a01acc5591fb8f4b38d087fa214613e0dbc822371b4ea7480442965 WatchSource:0}: Error finding container b31ffc499a01acc5591fb8f4b38d087fa214613e0dbc822371b4ea7480442965: Status 404 returned error can't find the container with id b31ffc499a01acc5591fb8f4b38d087fa214613e0dbc822371b4ea7480442965
Aug 01 01:09:44.116079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.116050    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258244    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-socket\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258337    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-envoy\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258377    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-data\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258439    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-credential-socket\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258473    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7300acfe-7970-4061-a187-239a7107d83f-istio-podinfo\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258519    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxsfr\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-kube-api-access-bxsfr\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258558    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-istio-token\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258640    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-certs\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.258672    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7300acfe-7970-4061-a187-239a7107d83f-istiod-ca-cert\") pod \"7300acfe-7970-4061-a187-239a7107d83f\" (UID: \"7300acfe-7970-4061-a187-239a7107d83f\") "
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.259529    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7300acfe-7970-4061-a187-239a7107d83f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:09:44.262875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.259828    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:44.264889 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.264646    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-data" (OuterVolumeSpecName: "istio-data") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:44.265055 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.264905    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:44.266086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.266058    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:44.267293 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.267262    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:09:44.267501 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.267475    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:09:44.268310 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.268276    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7300acfe-7970-4061-a187-239a7107d83f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:09:44.268554 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.268530    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-kube-api-access-bxsfr" (OuterVolumeSpecName: "kube-api-access-bxsfr") pod "7300acfe-7970-4061-a187-239a7107d83f" (UID: "7300acfe-7970-4061-a187-239a7107d83f"). InnerVolumeSpecName "kube-api-access-bxsfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:09:44.360257 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360219    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.360257 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360256    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.360257 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360270    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.360598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360283    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.360598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360299    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7300acfe-7970-4061-a187-239a7107d83f-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.360598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360311    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxsfr\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-kube-api-access-bxsfr\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.360598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360325    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7300acfe-7970-4061-a187-239a7107d83f-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.360598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360336    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7300acfe-7970-4061-a187-239a7107d83f-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.360598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.360347    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7300acfe-7970-4061-a187-239a7107d83f-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:09:44.501569 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.501538    2570 generic.go:356] "Generic (PLEG): container finished" podID="7300acfe-7970-4061-a187-239a7107d83f" containerID="ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688" exitCode=0
Aug 01 01:09:44.501723 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.501625    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"
Aug 01 01:09:44.501723 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.501636    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" event={"ID":"7300acfe-7970-4061-a187-239a7107d83f","Type":"ContainerDied","Data":"ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688"}
Aug 01 01:09:44.501723 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.501698    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9" event={"ID":"7300acfe-7970-4061-a187-239a7107d83f","Type":"ContainerDied","Data":"1e7e96ce0cf56e51ad4f80a665e4a5f5411038d81184c0e7d97fd5de101547ad"}
Aug 01 01:09:44.501901 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.501731    2570 scope.go:117] "RemoveContainer" containerID="ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688"
Aug 01 01:09:44.503040 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.503002    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x" event={"ID":"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7","Type":"ContainerStarted","Data":"b31ffc499a01acc5591fb8f4b38d087fa214613e0dbc822371b4ea7480442965"}
Aug 01 01:09:44.510567 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.510543    2570 scope.go:117] "RemoveContainer" containerID="11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb"
Aug 01 01:09:44.519583 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.519515    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"]
Aug 01 01:09:44.520136 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.520116    2570 scope.go:117] "RemoveContainer" containerID="ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688"
Aug 01 01:09:44.520441 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:44.520414    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688\": container with ID starting with ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688 not found: ID does not exist" containerID="ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688"
Aug 01 01:09:44.520494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.520455    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688"} err="failed to get container status \"ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688\": rpc error: code = NotFound desc = could not find container \"ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688\": container with ID starting with ff759b999b25208f0f8ea75c849c80baf9fae3f10af0be00418d34b74f922688 not found: ID does not exist"
Aug 01 01:09:44.520494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.520483    2570 scope.go:117] "RemoveContainer" containerID="11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb"
Aug 01 01:09:44.520808 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:09:44.520784    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb\": container with ID starting with 11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb not found: ID does not exist" containerID="11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb"
Aug 01 01:09:44.520866 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.520816    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb"} err="failed to get container status \"11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb\": rpc error: code = NotFound desc = could not find container \"11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb\": container with ID starting with 11344a5b242dc4b8506a2ff31273c11c94d66dddb703b81b6b6fb9d242c2eafb not found: ID does not exist"
Aug 01 01:09:44.521448 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:44.521430    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8e664d6c-openshift-default-66fc4f65cb-rbpm9"]
Aug 01 01:09:45.510225 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:45.510188    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x" event={"ID":"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7","Type":"ContainerStarted","Data":"1c31869594c1079fbfe12ba5205cc0ac0e5f1a9de217215589b39a3a568ee155"}
Aug 01 01:09:45.510629 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:45.510272    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:45.527632 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:45.527580    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x" podStartSLOduration=3.111005802 podStartE2EDuration="3.527561882s" totalImagesPullingTime="416.55608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:45.525121456 +0000 UTC m=+2151.759053739" watchObservedRunningTime="2026-08-01 01:09:45.527561882 +0000 UTC m=+2151.761494150"
Aug 01 01:09:46.262857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:46.262775    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7300acfe-7970-4061-a187-239a7107d83f" path="/var/lib/kubelet/pods/7300acfe-7970-4061-a187-239a7107d83f/volumes"
Aug 01 01:09:51.521351 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:51.521317    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:09:55.560056 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:55.560019    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerID="d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d" exitCode=0
Aug 01 01:09:55.560485 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:55.560098    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" event={"ID":"dc8efb4b-9be1-4b69-9a7e-ea842093deb4","Type":"ContainerDied","Data":"d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d"}
Aug 01 01:09:55.560531 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:55.560515    2570 scope.go:117] "RemoveContainer" containerID="d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d"
Aug 01 01:09:55.561597 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:55.561569    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:55.561704 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:55.561685    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:55.561745 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:55.561735    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:09:56.569874 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:56.569829    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" event={"ID":"dc8efb4b-9be1-4b69-9a7e-ea842093deb4","Type":"ContainerStarted","Data":"593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3"}
Aug 01 01:09:56.980400 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:56.980298    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:56.981700 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:56.981666    2570 patch_prober.go:28] interesting pod/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Aug 01 01:09:56.981853 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:56.981734    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Aug 01 01:09:57.488342 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:57.488300    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:09:57.981224 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:57.981187    2570 patch_prober.go:28] interesting pod/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Aug 01 01:09:57.981650 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:57.981243    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Aug 01 01:09:58.980753 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:58.980709    2570 patch_prober.go:28] interesting pod/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Aug 01 01:09:58.980985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:58.980773    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Aug 01 01:09:59.981339 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:59.981294    2570 patch_prober.go:28] interesting pod/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Aug 01 01:09:59.981800 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:09:59.981371    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Aug 01 01:10:00.981170 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:00.981124    2570 patch_prober.go:28] interesting pod/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Aug 01 01:10:00.981370 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:00.981206    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Aug 01 01:10:01.985511 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:01.985480    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:10:01.986423 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:01.986404    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:10:06.769155 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.769120    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"]
Aug 01 01:10:06.769651 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.769485    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy"
Aug 01 01:10:06.769651 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.769498    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy"
Aug 01 01:10:06.769651 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.769581    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy"
Aug 01 01:10:06.769651 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.769591    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy"
Aug 01 01:10:06.772684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.772660    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"
Aug 01 01:10:06.775176 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.775156    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\""
Aug 01 01:10:06.779520 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.779497    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"]
Aug 01 01:10:06.858843 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.858807    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63504d7a-bcd2-4e80-95fa-489a92effa2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg\" (UID: \"63504d7a-bcd2-4e80-95fa-489a92effa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"
Aug 01 01:10:06.959911 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.959874    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63504d7a-bcd2-4e80-95fa-489a92effa2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg\" (UID: \"63504d7a-bcd2-4e80-95fa-489a92effa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"
Aug 01 01:10:06.960503 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:06.960481    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63504d7a-bcd2-4e80-95fa-489a92effa2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg\" (UID: \"63504d7a-bcd2-4e80-95fa-489a92effa2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"
Aug 01 01:10:07.084511 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:07.084486    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"
Aug 01 01:10:07.214338 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:07.214271    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"]
Aug 01 01:10:07.216442 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:10:07.216413    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63504d7a_bcd2_4e80_95fa_489a92effa2d.slice/crio-4d8ab6d9ed728ee4280c955c2489bb3e7503345f0ea16db037b4cff4d89640d2 WatchSource:0}: Error finding container 4d8ab6d9ed728ee4280c955c2489bb3e7503345f0ea16db037b4cff4d89640d2: Status 404 returned error can't find the container with id 4d8ab6d9ed728ee4280c955c2489bb3e7503345f0ea16db037b4cff4d89640d2
Aug 01 01:10:07.625088 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:07.625056    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg" event={"ID":"63504d7a-bcd2-4e80-95fa-489a92effa2d","Type":"ContainerStarted","Data":"5df24d4f90120e6ab6b03a727c449d68c44186dd693306b8c6ef34f0e9db9ce1"}
Aug 01 01:10:07.625253 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:07.625115    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg" event={"ID":"63504d7a-bcd2-4e80-95fa-489a92effa2d","Type":"ContainerStarted","Data":"4d8ab6d9ed728ee4280c955c2489bb3e7503345f0ea16db037b4cff4d89640d2"}
Aug 01 01:10:07.640719 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:07.640671    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg" podStartSLOduration=1.640656634 podStartE2EDuration="1.640656634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:10:07.638727004 +0000 UTC m=+2173.872659271" watchObservedRunningTime="2026-08-01 01:10:07.640656634 +0000 UTC m=+2173.874588899"
Aug 01 01:10:08.631085 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:08.631050    2570 generic.go:356] "Generic (PLEG): container finished" podID="63504d7a-bcd2-4e80-95fa-489a92effa2d" containerID="5df24d4f90120e6ab6b03a727c449d68c44186dd693306b8c6ef34f0e9db9ce1" exitCode=0
Aug 01 01:10:08.631514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:08.631134    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg" event={"ID":"63504d7a-bcd2-4e80-95fa-489a92effa2d","Type":"ContainerDied","Data":"5df24d4f90120e6ab6b03a727c449d68c44186dd693306b8c6ef34f0e9db9ce1"}
Aug 01 01:10:09.768954 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:09.768914    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"
Aug 01 01:10:09.885235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:09.885199    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63504d7a-bcd2-4e80-95fa-489a92effa2d-maas-api-service-ca\") pod \"63504d7a-bcd2-4e80-95fa-489a92effa2d\" (UID: \"63504d7a-bcd2-4e80-95fa-489a92effa2d\") "
Aug 01 01:10:09.885570 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:09.885546    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63504d7a-bcd2-4e80-95fa-489a92effa2d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63504d7a-bcd2-4e80-95fa-489a92effa2d" (UID: "63504d7a-bcd2-4e80-95fa-489a92effa2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:10:09.986457 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:09.986382    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63504d7a-bcd2-4e80-95fa-489a92effa2d-maas-api-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:10.641857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:10.641824    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"
Aug 01 01:10:10.641857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:10.641844    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg" event={"ID":"63504d7a-bcd2-4e80-95fa-489a92effa2d","Type":"ContainerDied","Data":"4d8ab6d9ed728ee4280c955c2489bb3e7503345f0ea16db037b4cff4d89640d2"}
Aug 01 01:10:10.642105 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:10.641900    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d8ab6d9ed728ee4280c955c2489bb3e7503345f0ea16db037b4cff4d89640d2"
Aug 01 01:10:13.502373 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.502337    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"]
Aug 01 01:10:13.502787 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.502577    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x" podUID="6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" containerName="maas-api" containerID="cri-o://1c31869594c1079fbfe12ba5205cc0ac0e5f1a9de217215589b39a3a568ee155" gracePeriod=30
Aug 01 01:10:13.659134 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.659102    2570 generic.go:356] "Generic (PLEG): container finished" podID="6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" containerID="1c31869594c1079fbfe12ba5205cc0ac0e5f1a9de217215589b39a3a568ee155" exitCode=0
Aug 01 01:10:13.659304 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.659166    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x" event={"ID":"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7","Type":"ContainerDied","Data":"1c31869594c1079fbfe12ba5205cc0ac0e5f1a9de217215589b39a3a568ee155"}
Aug 01 01:10:13.674444 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.674377    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"]
Aug 01 01:10:13.676057 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.676030    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef000e4f-7pkhg"]
Aug 01 01:10:13.765944 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.765868    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:10:13.810231 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.810197    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"]
Aug 01 01:10:13.810451 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.810421    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy" containerID="cri-o://593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3" gracePeriod=30
Aug 01 01:10:13.819129 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.819100    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls\") pod \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") "
Aug 01 01:10:13.819259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.819156    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-openshift-service-ca\") pod \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") "
Aug 01 01:10:13.819259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.819216    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgqg9\" (UniqueName: \"kubernetes.io/projected/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-kube-api-access-dgqg9\") pod \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\" (UID: \"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7\") "
Aug 01 01:10:13.819626 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.819598    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" (UID: "6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:10:13.822049 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.822015    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-kube-api-access-dgqg9" (OuterVolumeSpecName: "kube-api-access-dgqg9") pod "6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" (UID: "6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7"). InnerVolumeSpecName "kube-api-access-dgqg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:10:13.822163 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.822080    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" (UID: "6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:10:13.920051 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.920021    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:13.920051 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.920049    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgqg9\" (UniqueName: \"kubernetes.io/projected/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-kube-api-access-dgqg9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:13.920245 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:13.920061    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:14.261914 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:14.261884    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63504d7a-bcd2-4e80-95fa-489a92effa2d" path="/var/lib/kubelet/pods/63504d7a-bcd2-4e80-95fa-489a92effa2d/volumes"
Aug 01 01:10:14.665054 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:14.665010    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x" event={"ID":"6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7","Type":"ContainerDied","Data":"b31ffc499a01acc5591fb8f4b38d087fa214613e0dbc822371b4ea7480442965"}
Aug 01 01:10:14.665054 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:14.665053    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"
Aug 01 01:10:14.665561 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:14.665081    2570 scope.go:117] "RemoveContainer" containerID="1c31869594c1079fbfe12ba5205cc0ac0e5f1a9de217215589b39a3a568ee155"
Aug 01 01:10:14.681621 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:14.681595    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"]
Aug 01 01:10:14.685074 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:14.685054    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef000e4f-f548b66b-6mz4x"]
Aug 01 01:10:16.262408 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:16.262375    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" path="/var/lib/kubelet/pods/6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7/volumes"
Aug 01 01:10:16.988851 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:16.988810    2570 patch_prober.go:28] interesting pod/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Aug 01 01:10:16.989049 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:16.988874    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 01:10:19.053553 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.053529    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:10:19.164612 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164544    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istiod-ca-cert\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.164612 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164585    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-certs\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.164612 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164603    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-data\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.164847 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164638    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-podinfo\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.164847 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164693    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-envoy\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.164847 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164730    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-token\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.164847 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164771    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69sdt\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-kube-api-access-69sdt\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.164847 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164804    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-credential-socket\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.165135 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164852    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-socket\") pod \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\" (UID: \"dc8efb4b-9be1-4b69-9a7e-ea842093deb4\") "
Aug 01 01:10:19.165135 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164864    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:10:19.165135 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.164908    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:10:19.165135 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.165067    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:10:19.165356 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.165151    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.165356 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.165170    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.165356 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.165184    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.165356 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.165280    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:10:19.165356 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.165307    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:10:19.167030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.166993    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:10:19.167154 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.167074    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-kube-api-access-69sdt" (OuterVolumeSpecName: "kube-api-access-69sdt") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "kube-api-access-69sdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:10:19.167264 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.167238    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:10:19.167334 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.167317    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc8efb4b-9be1-4b69-9a7e-ea842093deb4" (UID: "dc8efb4b-9be1-4b69-9a7e-ea842093deb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:10:19.265563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.265533    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.265563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.265557    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.265563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.265566    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.265877 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.265575    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69sdt\" (UniqueName: \"kubernetes.io/projected/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-kube-api-access-69sdt\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.265877 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.265586    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.265877 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.265595    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8efb4b-9be1-4b69-9a7e-ea842093deb4-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:10:19.697602 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.697568    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerID="593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3" exitCode=0
Aug 01 01:10:19.697759 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.697620    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"
Aug 01 01:10:19.697759 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.697647    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" event={"ID":"dc8efb4b-9be1-4b69-9a7e-ea842093deb4","Type":"ContainerDied","Data":"593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3"}
Aug 01 01:10:19.697759 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.697701    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9" event={"ID":"dc8efb4b-9be1-4b69-9a7e-ea842093deb4","Type":"ContainerDied","Data":"6458558330af52e07feb9520f3dfe262ea9cbdc44d1862987611d0fb8b06a772"}
Aug 01 01:10:19.697759 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.697723    2570 scope.go:117] "RemoveContainer" containerID="593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3"
Aug 01 01:10:19.709337 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.709320    2570 scope.go:117] "RemoveContainer" containerID="d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d"
Aug 01 01:10:19.717087 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.717051    2570 scope.go:117] "RemoveContainer" containerID="593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3"
Aug 01 01:10:19.717332 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:10:19.717313    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3\": container with ID starting with 593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3 not found: ID does not exist" containerID="593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3"
Aug 01 01:10:19.717419 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.717343    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3"} err="failed to get container status \"593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3\": rpc error: code = NotFound desc = could not find container \"593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3\": container with ID starting with 593c9ad0bbfe3cfee2afbb8567a02958f79f9fd6708cd528b4f875fa442d69a3 not found: ID does not exist"
Aug 01 01:10:19.717419 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.717368    2570 scope.go:117] "RemoveContainer" containerID="d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d"
Aug 01 01:10:19.717686 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:10:19.717661    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d\": container with ID starting with d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d not found: ID does not exist" containerID="d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d"
Aug 01 01:10:19.717811 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.717694    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d"} err="failed to get container status \"d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d\": rpc error: code = NotFound desc = could not find container \"d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d\": container with ID starting with d00b0c4b7711658616cc8abcf8b806640f10c9b92bb6b1426f7e9df694c7db2d not found: ID does not exist"
Aug 01 01:10:19.719706 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.719688    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"]
Aug 01 01:10:19.724300 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:19.724279    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ef000e4f-openshift-default-85b585cbd-hgjk9"]
Aug 01 01:10:20.261550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:10:20.261519    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" path="/var/lib/kubelet/pods/dc8efb4b-9be1-4b69-9a7e-ea842093deb4/volumes"
Aug 01 01:12:10.512123 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512087    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"]
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512420    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63504d7a-bcd2-4e80-95fa-489a92effa2d" containerName="revoke-keys"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512433    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="63504d7a-bcd2-4e80-95fa-489a92effa2d" containerName="revoke-keys"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512443    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512448    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512455    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512460    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7300acfe-7970-4061-a187-239a7107d83f" containerName="istio-proxy"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512469    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" containerName="maas-api"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512474    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" containerName="maas-api"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512483    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512488    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512546    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512553    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e09d91f-dcf3-41bf-8f93-3bc6dab0cff7" containerName="maas-api"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512559    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="63504d7a-bcd2-4e80-95fa-489a92effa2d" containerName="revoke-keys"
Aug 01 01:12:10.512601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.512565    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc8efb4b-9be1-4b69-9a7e-ea842093deb4" containerName="istio-proxy"
Aug 01 01:12:10.515396 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.515379    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.518031 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.518005    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Aug 01 01:12:10.518175 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.518036    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-86872dcf-openshift-default-dockercfg-k4ln2\""
Aug 01 01:12:10.534598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.534574    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"]
Aug 01 01:12:10.563547 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563522    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-podinfo\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.563658 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-socket\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.563658 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563629    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-data\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.563658 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563650    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqsv\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-kube-api-access-zfqsv\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.563776 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563677    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-certs\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.563776 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-token\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.563776 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563729    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-credential-socket\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.563776 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-envoy\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.563776 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.563763    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istiod-ca-cert\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.664883 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.664850    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-credential-socket\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.664883 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.664883    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-envoy\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665088 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.664905    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istiod-ca-cert\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665088 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.664956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-podinfo\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665088 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665009    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-socket\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665088 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665047    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-data\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665088 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqsv\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-kube-api-access-zfqsv\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665329 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665125    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-certs\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665430 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665402    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-credential-socket\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665611 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665470    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-socket\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665611 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665604    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-certs\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665611 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665506    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-token\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665827 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665541    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-data\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.665827 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.665777    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istiod-ca-cert\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.667714 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.667693    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-envoy\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.667941 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.667912    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-podinfo\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.672989 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.672970    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-token\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.673099 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.673035    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqsv\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-kube-api-access-zfqsv\") pod \"e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") " pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.825703 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.825671    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:10.967667 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.967637    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"]
Aug 01 01:12:10.970385 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:12:10.970352    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bf424f_1af1_4a9d_9e4a_612f153b6b12.slice/crio-17975f7ccfaad645d1ed9010514853376931360e3415d18ca68887e5605e9072 WatchSource:0}: Error finding container 17975f7ccfaad645d1ed9010514853376931360e3415d18ca68887e5605e9072: Status 404 returned error can't find the container with id 17975f7ccfaad645d1ed9010514853376931360e3415d18ca68887e5605e9072
Aug 01 01:12:10.972744 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.972713    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:12:10.972839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.972781    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:12:10.972839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:10.972810    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:12:11.213422 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:11.213323    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" event={"ID":"e0bf424f-1af1-4a9d-9e4a-612f153b6b12","Type":"ContainerStarted","Data":"b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360"}
Aug 01 01:12:11.213422 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:11.213383    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" event={"ID":"e0bf424f-1af1-4a9d-9e4a-612f153b6b12","Type":"ContainerStarted","Data":"17975f7ccfaad645d1ed9010514853376931360e3415d18ca68887e5605e9072"}
Aug 01 01:12:11.232307 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:11.232262    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" podStartSLOduration=1.232247249 podStartE2EDuration="1.232247249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:11.231811951 +0000 UTC m=+2297.465744217" watchObservedRunningTime="2026-08-01 01:12:11.232247249 +0000 UTC m=+2297.466179578"
Aug 01 01:12:11.826458 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:11.826415    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:12.831212 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:12.831180    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:13.223005 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:13.222891    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:13.223863 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:13.223842    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:14.501512 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.501476    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"]
Aug 01 01:12:14.504790 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.504773    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.507328 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.507304    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-86872dcf-serving-cert\""
Aug 01 01:12:14.515634 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.515610    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"]
Aug 01 01:12:14.591952 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.591892    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zjm\" (UniqueName: \"kubernetes.io/projected/67e5c869-2cda-40a0-adac-19d2cf1b223d-kube-api-access-q2zjm\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.592119 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.591960    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e5c869-2cda-40a0-adac-19d2cf1b223d-maas-api-tls\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.592119 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.592018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c869-2cda-40a0-adac-19d2cf1b223d-openshift-service-ca\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.692652 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.692613    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zjm\" (UniqueName: \"kubernetes.io/projected/67e5c869-2cda-40a0-adac-19d2cf1b223d-kube-api-access-q2zjm\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.692652 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.692658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e5c869-2cda-40a0-adac-19d2cf1b223d-maas-api-tls\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.692949 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.692704    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c869-2cda-40a0-adac-19d2cf1b223d-openshift-service-ca\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.693361 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.693335    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c869-2cda-40a0-adac-19d2cf1b223d-openshift-service-ca\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.695132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.695111    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e5c869-2cda-40a0-adac-19d2cf1b223d-maas-api-tls\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.700316 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.700292    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zjm\" (UniqueName: \"kubernetes.io/projected/67e5c869-2cda-40a0-adac-19d2cf1b223d-kube-api-access-q2zjm\") pod \"maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.817805 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.817769    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:14.967015 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:14.966982    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"]
Aug 01 01:12:14.970356 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:12:14.970327    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e5c869_2cda_40a0_adac_19d2cf1b223d.slice/crio-a35e21e25e133771d5aa65092a13d24a5d2093b5c2302075d80195ea603549b3 WatchSource:0}: Error finding container a35e21e25e133771d5aa65092a13d24a5d2093b5c2302075d80195ea603549b3: Status 404 returned error can't find the container with id a35e21e25e133771d5aa65092a13d24a5d2093b5c2302075d80195ea603549b3
Aug 01 01:12:15.152524 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.152444    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"]
Aug 01 01:12:15.156995 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.156978    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:15.159561 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.159532    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-86872dcf-dockercfg-4rxkw\""
Aug 01 01:12:15.159669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.159571    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-86872dcf\""
Aug 01 01:12:15.165214 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.165188    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"]
Aug 01 01:12:15.192364 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.192333    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"]
Aug 01 01:12:15.195596 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.195577    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:15.197225 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.197202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c17347-e33f-426d-866b-24591747e08a-plugins-config-volume\") pod \"payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz\" (UID: \"d9c17347-e33f-426d-866b-24591747e08a\") " pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:15.197305 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.197272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrw7\" (UniqueName: \"kubernetes.io/projected/d9c17347-e33f-426d-866b-24591747e08a-kube-api-access-7qrw7\") pod \"payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz\" (UID: \"d9c17347-e33f-426d-866b-24591747e08a\") " pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:15.205030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.204997    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"]
Aug 01 01:12:15.233174 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.233144    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w" event={"ID":"67e5c869-2cda-40a0-adac-19d2cf1b223d","Type":"ContainerStarted","Data":"a35e21e25e133771d5aa65092a13d24a5d2093b5c2302075d80195ea603549b3"}
Aug 01 01:12:15.297652 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.297617    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c17347-e33f-426d-866b-24591747e08a-plugins-config-volume\") pod \"payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz\" (UID: \"d9c17347-e33f-426d-866b-24591747e08a\") " pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:15.297832 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.297670    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz\" (UID: \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:15.297832 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.297773    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2bs\" (UniqueName: \"kubernetes.io/projected/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-kube-api-access-mj2bs\") pod \"payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz\" (UID: \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:15.297977 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.297830    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrw7\" (UniqueName: \"kubernetes.io/projected/d9c17347-e33f-426d-866b-24591747e08a-kube-api-access-7qrw7\") pod \"payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz\" (UID: \"d9c17347-e33f-426d-866b-24591747e08a\") " pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:15.298489 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.298470    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c17347-e33f-426d-866b-24591747e08a-plugins-config-volume\") pod \"payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz\" (UID: \"d9c17347-e33f-426d-866b-24591747e08a\") " pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:15.305513 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.305488    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrw7\" (UniqueName: \"kubernetes.io/projected/d9c17347-e33f-426d-866b-24591747e08a-kube-api-access-7qrw7\") pod \"payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz\" (UID: \"d9c17347-e33f-426d-866b-24591747e08a\") " pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:15.399472 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.399450    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz\" (UID: \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:15.399579 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.399492    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2bs\" (UniqueName: \"kubernetes.io/projected/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-kube-api-access-mj2bs\") pod \"payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz\" (UID: \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:15.400127 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.400109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz\" (UID: \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:15.410286 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.410229    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2bs\" (UniqueName: \"kubernetes.io/projected/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-kube-api-access-mj2bs\") pod \"payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz\" (UID: \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:15.468607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.468576    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:15.506022 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.505995    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:15.616476 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.616445    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"]
Aug 01 01:12:15.619169 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:12:15.619139    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c17347_e33f_426d_866b_24591747e08a.slice/crio-2aa8950a8d6b0e78eb0e1fc5b6763a25c16f5dc39a43c658906b15bdf22ee86d WatchSource:0}: Error finding container 2aa8950a8d6b0e78eb0e1fc5b6763a25c16f5dc39a43c658906b15bdf22ee86d: Status 404 returned error can't find the container with id 2aa8950a8d6b0e78eb0e1fc5b6763a25c16f5dc39a43c658906b15bdf22ee86d
Aug 01 01:12:15.655282 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:15.655257    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"]
Aug 01 01:12:15.659013 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:12:15.658981    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c2b5c0_49d2_4dbe_8d26_30e96a94fca7.slice/crio-bee6562280e37ed4fe401293602bde7bf5acbdada24c8b9d4f8fba625064f8fc WatchSource:0}: Error finding container bee6562280e37ed4fe401293602bde7bf5acbdada24c8b9d4f8fba625064f8fc: Status 404 returned error can't find the container with id bee6562280e37ed4fe401293602bde7bf5acbdada24c8b9d4f8fba625064f8fc
Aug 01 01:12:16.239547 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.239511    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w" event={"ID":"67e5c869-2cda-40a0-adac-19d2cf1b223d","Type":"ContainerStarted","Data":"b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28"}
Aug 01 01:12:16.239750 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.239686    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:16.241128 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.241094    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz" event={"ID":"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7","Type":"ContainerStarted","Data":"c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346"}
Aug 01 01:12:16.241260 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.241156    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz" event={"ID":"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7","Type":"ContainerStarted","Data":"bee6562280e37ed4fe401293602bde7bf5acbdada24c8b9d4f8fba625064f8fc"}
Aug 01 01:12:16.241260 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.241207    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:16.242625 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.242603    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz" event={"ID":"d9c17347-e33f-426d-866b-24591747e08a","Type":"ContainerStarted","Data":"43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5"}
Aug 01 01:12:16.242784 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.242641    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz" event={"ID":"d9c17347-e33f-426d-866b-24591747e08a","Type":"ContainerStarted","Data":"2aa8950a8d6b0e78eb0e1fc5b6763a25c16f5dc39a43c658906b15bdf22ee86d"}
Aug 01 01:12:16.242784 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.242732    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:16.256567 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.256518    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w" podStartSLOduration=1.838967327 podStartE2EDuration="2.256500714s" totalImagesPullingTime="417.533387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:16.255160941 +0000 UTC m=+2302.489093208" watchObservedRunningTime="2026-08-01 01:12:16.256500714 +0000 UTC m=+2302.490432982"
Aug 01 01:12:16.270862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.270824    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz" podStartSLOduration=1.270812649 podStartE2EDuration="1.270812649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:16.269853637 +0000 UTC m=+2302.503785907" watchObservedRunningTime="2026-08-01 01:12:16.270812649 +0000 UTC m=+2302.504744914"
Aug 01 01:12:16.285728 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:16.285684    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz" podStartSLOduration=1.285673024 podStartE2EDuration="1.285673024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:12:16.284037673 +0000 UTC m=+2302.517969940" watchObservedRunningTime="2026-08-01 01:12:16.285673024 +0000 UTC m=+2302.519605289"
Aug 01 01:12:22.254822 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:22.254789    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:27.249676 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:27.249583    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:27.249676 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:27.249657    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:27.309729 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:27.309693    2570 generic.go:356] "Generic (PLEG): container finished" podID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerID="b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360" exitCode=0
Aug 01 01:12:27.309883 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:27.309761    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" event={"ID":"e0bf424f-1af1-4a9d-9e4a-612f153b6b12","Type":"ContainerDied","Data":"b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360"}
Aug 01 01:12:27.310144 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:27.310129    2570 scope.go:117] "RemoveContainer" containerID="b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360"
Aug 01 01:12:27.311182 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:27.311152    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:12:27.311298 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:27.311230    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:12:27.311298 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:27.311279    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:12:28.222974 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:28.222920    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:28.317257 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:28.317224    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" event={"ID":"e0bf424f-1af1-4a9d-9e4a-612f153b6b12","Type":"ContainerStarted","Data":"2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448"}
Aug 01 01:12:28.826875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:28.826836    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:28.828023 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:28.828002    2570 patch_prober.go:28] interesting pod/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Aug 01 01:12:28.828141 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:28.828046    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Aug 01 01:12:29.826791 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:29.826753    2570 patch_prober.go:28] interesting pod/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Aug 01 01:12:29.827232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:29.826810    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Aug 01 01:12:30.826583 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:30.826549    2570 patch_prober.go:28] interesting pod/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Aug 01 01:12:30.826767 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:30.826607    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Aug 01 01:12:31.827080 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:31.827037    2570 patch_prober.go:28] interesting pod/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Aug 01 01:12:31.827503 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:31.827110    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Aug 01 01:12:32.826799 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:32.826760    2570 patch_prober.go:28] interesting pod/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Aug 01 01:12:32.827024 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:32.826823    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Aug 01 01:12:33.831323 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:33.831290    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:33.831811 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:33.831582    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:33.832374 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:33.832352    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:12:55.943785 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:55.943573    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"]
Aug 01 01:12:55.944373 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:55.943832    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w" podUID="67e5c869-2cda-40a0-adac-19d2cf1b223d" containerName="maas-api" containerID="cri-o://b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28" gracePeriod=30
Aug 01 01:12:55.990848 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:55.990812    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"]
Aug 01 01:12:55.991149 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:55.991094    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz" podUID="d9c17347-e33f-426d-866b-24591747e08a" containerName="payload-processing" containerID="cri-o://43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5" gracePeriod=30
Aug 01 01:12:55.998687 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:55.998462    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"]
Aug 01 01:12:55.998774 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:55.998676    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz" podUID="22c2b5c0-49d2-4dbe-8d26-30e96a94fca7" containerName="payload-pre-processing" containerID="cri-o://c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346" gracePeriod=30
Aug 01 01:12:56.253646 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.253619    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:56.256886 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.256863    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:56.303015 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.302984    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:56.360827 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.360795    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2zjm\" (UniqueName: \"kubernetes.io/projected/67e5c869-2cda-40a0-adac-19d2cf1b223d-kube-api-access-q2zjm\") pod \"67e5c869-2cda-40a0-adac-19d2cf1b223d\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") "
Aug 01 01:12:56.361036 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.360884    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c17347-e33f-426d-866b-24591747e08a-plugins-config-volume\") pod \"d9c17347-e33f-426d-866b-24591747e08a\" (UID: \"d9c17347-e33f-426d-866b-24591747e08a\") "
Aug 01 01:12:56.361036 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.360985    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qrw7\" (UniqueName: \"kubernetes.io/projected/d9c17347-e33f-426d-866b-24591747e08a-kube-api-access-7qrw7\") pod \"d9c17347-e33f-426d-866b-24591747e08a\" (UID: \"d9c17347-e33f-426d-866b-24591747e08a\") "
Aug 01 01:12:56.361183 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.361135    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c869-2cda-40a0-adac-19d2cf1b223d-openshift-service-ca\") pod \"67e5c869-2cda-40a0-adac-19d2cf1b223d\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") "
Aug 01 01:12:56.361255 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.361237    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e5c869-2cda-40a0-adac-19d2cf1b223d-maas-api-tls\") pod \"67e5c869-2cda-40a0-adac-19d2cf1b223d\" (UID: \"67e5c869-2cda-40a0-adac-19d2cf1b223d\") "
Aug 01 01:12:56.361367 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.361346    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c17347-e33f-426d-866b-24591747e08a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9c17347-e33f-426d-866b-24591747e08a" (UID: "d9c17347-e33f-426d-866b-24591747e08a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:12:56.361560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.361536    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c869-2cda-40a0-adac-19d2cf1b223d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67e5c869-2cda-40a0-adac-19d2cf1b223d" (UID: "67e5c869-2cda-40a0-adac-19d2cf1b223d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:12:56.361684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.361667    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c17347-e33f-426d-866b-24591747e08a-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:12:56.361756 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.361716    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e5c869-2cda-40a0-adac-19d2cf1b223d-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:12:56.363190 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.363170    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c17347-e33f-426d-866b-24591747e08a-kube-api-access-7qrw7" (OuterVolumeSpecName: "kube-api-access-7qrw7") pod "d9c17347-e33f-426d-866b-24591747e08a" (UID: "d9c17347-e33f-426d-866b-24591747e08a"). InnerVolumeSpecName "kube-api-access-7qrw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:12:56.363396 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.363374    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e5c869-2cda-40a0-adac-19d2cf1b223d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67e5c869-2cda-40a0-adac-19d2cf1b223d" (UID: "67e5c869-2cda-40a0-adac-19d2cf1b223d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:12:56.363440 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.363377    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e5c869-2cda-40a0-adac-19d2cf1b223d-kube-api-access-q2zjm" (OuterVolumeSpecName: "kube-api-access-q2zjm") pod "67e5c869-2cda-40a0-adac-19d2cf1b223d" (UID: "67e5c869-2cda-40a0-adac-19d2cf1b223d"). InnerVolumeSpecName "kube-api-access-q2zjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:12:56.457590 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.457562    2570 generic.go:356] "Generic (PLEG): container finished" podID="67e5c869-2cda-40a0-adac-19d2cf1b223d" containerID="b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28" exitCode=0
Aug 01 01:12:56.457711 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.457620    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"
Aug 01 01:12:56.457711 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.457635    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w" event={"ID":"67e5c869-2cda-40a0-adac-19d2cf1b223d","Type":"ContainerDied","Data":"b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28"}
Aug 01 01:12:56.457871 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.457727    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w" event={"ID":"67e5c869-2cda-40a0-adac-19d2cf1b223d","Type":"ContainerDied","Data":"a35e21e25e133771d5aa65092a13d24a5d2093b5c2302075d80195ea603549b3"}
Aug 01 01:12:56.457871 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.457760    2570 scope.go:117] "RemoveContainer" containerID="b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28"
Aug 01 01:12:56.459146 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.459124    2570 generic.go:356] "Generic (PLEG): container finished" podID="22c2b5c0-49d2-4dbe-8d26-30e96a94fca7" containerID="c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346" exitCode=0
Aug 01 01:12:56.459242 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.459158    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"
Aug 01 01:12:56.459242 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.459207    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz" event={"ID":"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7","Type":"ContainerDied","Data":"c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346"}
Aug 01 01:12:56.459353 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.459282    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz" event={"ID":"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7","Type":"ContainerDied","Data":"bee6562280e37ed4fe401293602bde7bf5acbdada24c8b9d4f8fba625064f8fc"}
Aug 01 01:12:56.460653 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.460633    2570 generic.go:356] "Generic (PLEG): container finished" podID="d9c17347-e33f-426d-866b-24591747e08a" containerID="43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5" exitCode=0
Aug 01 01:12:56.460754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.460664    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz" event={"ID":"d9c17347-e33f-426d-866b-24591747e08a","Type":"ContainerDied","Data":"43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5"}
Aug 01 01:12:56.460754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.460681    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"
Aug 01 01:12:56.460754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.460694    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz" event={"ID":"d9c17347-e33f-426d-866b-24591747e08a","Type":"ContainerDied","Data":"2aa8950a8d6b0e78eb0e1fc5b6763a25c16f5dc39a43c658906b15bdf22ee86d"}
Aug 01 01:12:56.462147 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.462095    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj2bs\" (UniqueName: \"kubernetes.io/projected/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-kube-api-access-mj2bs\") pod \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\" (UID: \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\") "
Aug 01 01:12:56.462147 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.462143    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-plugins-config-volume\") pod \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\" (UID: \"22c2b5c0-49d2-4dbe-8d26-30e96a94fca7\") "
Aug 01 01:12:56.462357 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.462339    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e5c869-2cda-40a0-adac-19d2cf1b223d-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:12:56.462357 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.462358    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2zjm\" (UniqueName: \"kubernetes.io/projected/67e5c869-2cda-40a0-adac-19d2cf1b223d-kube-api-access-q2zjm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:12:56.462558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.462369    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qrw7\" (UniqueName: \"kubernetes.io/projected/d9c17347-e33f-426d-866b-24591747e08a-kube-api-access-7qrw7\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:12:56.462558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.462542    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22c2b5c0-49d2-4dbe-8d26-30e96a94fca7" (UID: "22c2b5c0-49d2-4dbe-8d26-30e96a94fca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:12:56.464188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.464166    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-kube-api-access-mj2bs" (OuterVolumeSpecName: "kube-api-access-mj2bs") pod "22c2b5c0-49d2-4dbe-8d26-30e96a94fca7" (UID: "22c2b5c0-49d2-4dbe-8d26-30e96a94fca7"). InnerVolumeSpecName "kube-api-access-mj2bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:12:56.467762 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.467744    2570 scope.go:117] "RemoveContainer" containerID="b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28"
Aug 01 01:12:56.468037 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:12:56.468018    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28\": container with ID starting with b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28 not found: ID does not exist" containerID="b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28"
Aug 01 01:12:56.468137 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.468058    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28"} err="failed to get container status \"b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28\": rpc error: code = NotFound desc = could not find container \"b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28\": container with ID starting with b1ea0cf6fee2cc3b2c281cb2d50e924c202b546df9e111b811f4fcb0a3908b28 not found: ID does not exist"
Aug 01 01:12:56.468137 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.468081    2570 scope.go:117] "RemoveContainer" containerID="c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346"
Aug 01 01:12:56.477884 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.477868    2570 scope.go:117] "RemoveContainer" containerID="c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346"
Aug 01 01:12:56.478365 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:12:56.478336    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346\": container with ID starting with c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346 not found: ID does not exist" containerID="c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346"
Aug 01 01:12:56.478453 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.478372    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346"} err="failed to get container status \"c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346\": rpc error: code = NotFound desc = could not find container \"c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346\": container with ID starting with c2908263b5e2d507cd49b2cabc42a6b05c75581ae74236fde9fb522197f8f346 not found: ID does not exist"
Aug 01 01:12:56.478453 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.478388    2570 scope.go:117] "RemoveContainer" containerID="43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5"
Aug 01 01:12:56.482386 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.482365    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"]
Aug 01 01:12:56.486185 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.486166    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86872dcf-5bc5d845fd-4sv9w"]
Aug 01 01:12:56.486369 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.486356    2570 scope.go:117] "RemoveContainer" containerID="43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5"
Aug 01 01:12:56.486622 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:12:56.486600    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5\": container with ID starting with 43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5 not found: ID does not exist" containerID="43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5"
Aug 01 01:12:56.486699 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.486625    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5"} err="failed to get container status \"43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5\": rpc error: code = NotFound desc = could not find container \"43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5\": container with ID starting with 43512ce5545090d0f5a6e3fff17d55305319bffd81d6d9f55d91325156eb94b5 not found: ID does not exist"
Aug 01 01:12:56.495278 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.495257    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"]
Aug 01 01:12:56.500912 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.500893    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-86872dcf-66dcc6497d-d5hnz"]
Aug 01 01:12:56.563754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.563732    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2bs\" (UniqueName: \"kubernetes.io/projected/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-kube-api-access-mj2bs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:12:56.563754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.563752    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:12:56.780188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.780026    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"]
Aug 01 01:12:56.782008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:56.781985    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-86872dcf-564f865656-clfmz"]
Aug 01 01:12:58.262341 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:58.262307    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c2b5c0-49d2-4dbe-8d26-30e96a94fca7" path="/var/lib/kubelet/pods/22c2b5c0-49d2-4dbe-8d26-30e96a94fca7/volumes"
Aug 01 01:12:58.262717 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:58.262697    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e5c869-2cda-40a0-adac-19d2cf1b223d" path="/var/lib/kubelet/pods/67e5c869-2cda-40a0-adac-19d2cf1b223d/volumes"
Aug 01 01:12:58.263027 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:12:58.263014    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c17347-e33f-426d-866b-24591747e08a" path="/var/lib/kubelet/pods/d9c17347-e33f-426d-866b-24591747e08a/volumes"
Aug 01 01:13:02.614287 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:02.614252    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"]
Aug 01 01:13:02.614783 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:02.614513    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy" containerID="cri-o://2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448" gracePeriod=30
Aug 01 01:13:03.834817 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:03.834776    2570 patch_prober.go:28] interesting pod/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Aug 01 01:13:03.835233 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:03.834836    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 01:13:07.857490 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.857463    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:13:07.957080 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.956990    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfqsv\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-kube-api-access-zfqsv\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.957080 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.957027    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-token\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.958046 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.957607    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-podinfo\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.958046 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.957730    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-envoy\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.958046 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.957783    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-socket\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.958046 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.957823    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-credential-socket\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.958046 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.957956    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-certs\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.958046 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.958054    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istiod-ca-cert\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.958577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.958095    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-data\") pod \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\" (UID: \"e0bf424f-1af1-4a9d-9e4a-612f153b6b12\") "
Aug 01 01:13:07.958577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.958121    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:13:07.958577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.958333    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:13:07.958740 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.958612    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:07.958740 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.958632    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:07.958740 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.958687    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:13:07.961973 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.960082    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:13:07.961973 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.961162    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:13:07.962372 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.962345    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:13:07.966558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.966525    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:13:07.967955 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.967813    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:13:07.968020 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:07.967999    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-kube-api-access-zfqsv" (OuterVolumeSpecName: "kube-api-access-zfqsv") pod "e0bf424f-1af1-4a9d-9e4a-612f153b6b12" (UID: "e0bf424f-1af1-4a9d-9e4a-612f153b6b12"). InnerVolumeSpecName "kube-api-access-zfqsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:13:08.059259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.059218    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:08.059259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.059253    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:08.059259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.059265    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:08.059507 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.059276    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfqsv\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-kube-api-access-zfqsv\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:08.059507 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.059289    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:08.059507 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.059300    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:08.059507 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.059312    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bf424f-1af1-4a9d-9e4a-612f153b6b12-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:13:08.519650 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.519613    2570 generic.go:356] "Generic (PLEG): container finished" podID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerID="2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448" exitCode=0
Aug 01 01:13:08.519857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.519687    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"
Aug 01 01:13:08.519857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.519701    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" event={"ID":"e0bf424f-1af1-4a9d-9e4a-612f153b6b12","Type":"ContainerDied","Data":"2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448"}
Aug 01 01:13:08.519857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.519764    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v" event={"ID":"e0bf424f-1af1-4a9d-9e4a-612f153b6b12","Type":"ContainerDied","Data":"17975f7ccfaad645d1ed9010514853376931360e3415d18ca68887e5605e9072"}
Aug 01 01:13:08.519857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.519788    2570 scope.go:117] "RemoveContainer" containerID="2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448"
Aug 01 01:13:08.528320 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.528285    2570 scope.go:117] "RemoveContainer" containerID="b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360"
Aug 01 01:13:08.536524 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.536505    2570 scope.go:117] "RemoveContainer" containerID="2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448"
Aug 01 01:13:08.536785 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:13:08.536764    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448\": container with ID starting with 2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448 not found: ID does not exist" containerID="2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448"
Aug 01 01:13:08.536853 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.536792    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448"} err="failed to get container status \"2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448\": rpc error: code = NotFound desc = could not find container \"2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448\": container with ID starting with 2f737edded5b563c5b38ef191887446b37a631126fcfd64f65b3eb8017571448 not found: ID does not exist"
Aug 01 01:13:08.536853 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.536810    2570 scope.go:117] "RemoveContainer" containerID="b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360"
Aug 01 01:13:08.537061 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:13:08.537046    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360\": container with ID starting with b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360 not found: ID does not exist" containerID="b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360"
Aug 01 01:13:08.537108 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.537066    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360"} err="failed to get container status \"b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360\": rpc error: code = NotFound desc = could not find container \"b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360\": container with ID starting with b455d42578dbed7a9176d03b53b79b2b436feb7a88283054332b9143592fa360 not found: ID does not exist"
Aug 01 01:13:08.537178 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.537161    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"]
Aug 01 01:13:08.541231 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:08.541208    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-86872dcf-openshift-default-88947b8b8-mlq8v"]
Aug 01 01:13:10.262030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:13:10.261994    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" path="/var/lib/kubelet/pods/e0bf424f-1af1-4a9d-9e4a-612f153b6b12/volumes"
Aug 01 01:15:00.135222 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135177    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf"]
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135654    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c17347-e33f-426d-866b-24591747e08a" containerName="payload-processing"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135677    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c17347-e33f-426d-866b-24591747e08a" containerName="payload-processing"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135688    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c2b5c0-49d2-4dbe-8d26-30e96a94fca7" containerName="payload-pre-processing"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135698    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c2b5c0-49d2-4dbe-8d26-30e96a94fca7" containerName="payload-pre-processing"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135719    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135736    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135758    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c869-2cda-40a0-adac-19d2cf1b223d" containerName="maas-api"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135769    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c869-2cda-40a0-adac-19d2cf1b223d" containerName="maas-api"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135778    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy"
Aug 01 01:15:00.135862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135788    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy"
Aug 01 01:15:00.136415 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135879    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e5c869-2cda-40a0-adac-19d2cf1b223d" containerName="maas-api"
Aug 01 01:15:00.136415 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135892    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy"
Aug 01 01:15:00.136415 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135903    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c17347-e33f-426d-866b-24591747e08a" containerName="payload-processing"
Aug 01 01:15:00.136415 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135912    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bf424f-1af1-4a9d-9e4a-612f153b6b12" containerName="istio-proxy"
Aug 01 01:15:00.136415 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.135922    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c2b5c0-49d2-4dbe-8d26-30e96a94fca7" containerName="payload-pre-processing"
Aug 01 01:15:00.138994 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.138974    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf"
Aug 01 01:15:00.141318 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.141299    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\""
Aug 01 01:15:00.145004 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.144985    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf"]
Aug 01 01:15:00.150277 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.150254    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf"
Aug 01 01:15:00.294768 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:00.294743    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf"]
Aug 01 01:15:00.296641 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:15:00.296601    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816d33c7_b1e0_4de7_a170_824b1e3b9cdd.slice/crio-ec521d8b0ada68f29b4b9b400e7c944b5a3c00dc08b3e65c8128f86f13cc5e6b WatchSource:0}: Error finding container ec521d8b0ada68f29b4b9b400e7c944b5a3c00dc08b3e65c8128f86f13cc5e6b: Status 404 returned error can't find the container with id ec521d8b0ada68f29b4b9b400e7c944b5a3c00dc08b3e65c8128f86f13cc5e6b
Aug 01 01:15:01.031812 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:01.031771    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf" event={"ID":"816d33c7-b1e0-4de7-a170-824b1e3b9cdd","Type":"ContainerStarted","Data":"11da9c7ea7f2c95587914b31e8f15a56abad565ff77048a8c4e78c587a954574"}
Aug 01 01:15:01.032050 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:01.031825    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf" event={"ID":"816d33c7-b1e0-4de7-a170-824b1e3b9cdd","Type":"ContainerStarted","Data":"ec521d8b0ada68f29b4b9b400e7c944b5a3c00dc08b3e65c8128f86f13cc5e6b"}
Aug 01 01:15:01.047683 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:01.047632    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf" podStartSLOduration=1.047617282 podStartE2EDuration="1.047617282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:15:01.046975389 +0000 UTC m=+2467.280907656" watchObservedRunningTime="2026-08-01 01:15:01.047617282 +0000 UTC m=+2467.281549549"
Aug 01 01:15:02.038068 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:02.038032    2570 generic.go:356] "Generic (PLEG): container finished" podID="816d33c7-b1e0-4de7-a170-824b1e3b9cdd" containerID="11da9c7ea7f2c95587914b31e8f15a56abad565ff77048a8c4e78c587a954574" exitCode=0
Aug 01 01:15:02.038451 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:02.038117    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf" event={"ID":"816d33c7-b1e0-4de7-a170-824b1e3b9cdd","Type":"ContainerDied","Data":"11da9c7ea7f2c95587914b31e8f15a56abad565ff77048a8c4e78c587a954574"}
Aug 01 01:15:03.169270 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:03.169238    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf"
Aug 01 01:15:04.047795 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:04.047763    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf"
Aug 01 01:15:04.048019 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:04.047756    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759115-sbjlf" event={"ID":"816d33c7-b1e0-4de7-a170-824b1e3b9cdd","Type":"ContainerDied","Data":"ec521d8b0ada68f29b4b9b400e7c944b5a3c00dc08b3e65c8128f86f13cc5e6b"}
Aug 01 01:15:04.048019 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:04.047898    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec521d8b0ada68f29b4b9b400e7c944b5a3c00dc08b3e65c8128f86f13cc5e6b"
Aug 01 01:15:17.772805 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.772767    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"]
Aug 01 01:15:17.773285 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.773173    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816d33c7-b1e0-4de7-a170-824b1e3b9cdd" containerName="cleanup"
Aug 01 01:15:17.773285 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.773189    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="816d33c7-b1e0-4de7-a170-824b1e3b9cdd" containerName="cleanup"
Aug 01 01:15:17.773285 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.773269    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="816d33c7-b1e0-4de7-a170-824b1e3b9cdd" containerName="cleanup"
Aug 01 01:15:17.776302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.776283    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.779112 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.779089    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Aug 01 01:15:17.779213 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.779108    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ff21edf2-openshift-default-dockercfg-hnjxq\""
Aug 01 01:15:17.786826 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.786804    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"]
Aug 01 01:15:17.813786 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.813752    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-token\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.813990 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.813793    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istiod-ca-cert\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.813990 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.813851    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-podinfo\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.813990 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.813909    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-envoy\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.813990 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.813977    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-data\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.814194 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.814017    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-socket\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.814194 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.814041    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-certs\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.814194 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.814076    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2dk5\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-kube-api-access-m2dk5\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.814194 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.814109    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-credential-socket\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.914804 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.914767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istiod-ca-cert\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.914804 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.914810    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-podinfo\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.914836    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-envoy\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.914855    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-data\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.914889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-socket\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.914907    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-certs\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.914962    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2dk5\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-kube-api-access-m2dk5\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.914985    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-credential-socket\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.915039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-token\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915505 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.915322    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-data\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915624 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.915523    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-credential-socket\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915624 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.915604    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-socket\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915739 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.915621    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istiod-ca-cert\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.915739 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.915697    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-certs\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.917338 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.917312    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-envoy\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.917732 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.917700    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-podinfo\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.922822 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.922794    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2dk5\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-kube-api-access-m2dk5\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:17.922822 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:17.922804    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-token\") pod \"e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") " pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:18.087192 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:18.087150    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:18.223047 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:18.223015    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"]
Aug 01 01:15:18.224587 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:15:18.224560    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45884bcb_c6b4_47e9_acd4_166c5e45bf91.slice/crio-69fe477c8fa4d57373732e253a4aeae93051cd69356e89956579403c58441f84 WatchSource:0}: Error finding container 69fe477c8fa4d57373732e253a4aeae93051cd69356e89956579403c58441f84: Status 404 returned error can't find the container with id 69fe477c8fa4d57373732e253a4aeae93051cd69356e89956579403c58441f84
Aug 01 01:15:18.227140 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:18.227107    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:15:18.227214 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:18.227183    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:15:18.227214 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:18.227212    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:15:19.124563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:19.124520    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" event={"ID":"45884bcb-c6b4-47e9-acd4-166c5e45bf91","Type":"ContainerStarted","Data":"9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0"}
Aug 01 01:15:19.125010 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:19.124623    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" event={"ID":"45884bcb-c6b4-47e9-acd4-166c5e45bf91","Type":"ContainerStarted","Data":"69fe477c8fa4d57373732e253a4aeae93051cd69356e89956579403c58441f84"}
Aug 01 01:15:19.145209 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:19.145159    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" podStartSLOduration=2.145143269 podStartE2EDuration="2.145143269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:15:19.143278324 +0000 UTC m=+2485.377210614" watchObservedRunningTime="2026-08-01 01:15:19.145143269 +0000 UTC m=+2485.379075534"
Aug 01 01:15:20.087993 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:20.087954    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:20.094443 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:20.094417    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:20.130204 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:20.130171    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:20.131216 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:20.131195    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:22.167219 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.167187    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"]
Aug 01 01:15:22.170957 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.170923    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.173452 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.173421    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ff21edf2-serving-cert\""
Aug 01 01:15:22.181741 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.181719    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"]
Aug 01 01:15:22.255146 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.255109    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.255346 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.255300    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmsc\" (UniqueName: \"kubernetes.io/projected/6d62a22a-31b0-498c-b5e1-15886a2df122-kube-api-access-rnmsc\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.255416 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.255370    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d62a22a-31b0-498c-b5e1-15886a2df122-openshift-service-ca\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.356174 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.356141    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.356349 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.356226    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmsc\" (UniqueName: \"kubernetes.io/projected/6d62a22a-31b0-498c-b5e1-15886a2df122-kube-api-access-rnmsc\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.356349 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.356259    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d62a22a-31b0-498c-b5e1-15886a2df122-openshift-service-ca\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.356349 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:15:22.356282    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-serving-cert: secret "maas-api-e2e-mt-ff21edf2-serving-cert" not found
Aug 01 01:15:22.356349 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:15:22.356346    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls podName:6d62a22a-31b0-498c-b5e1-15886a2df122 nodeName:}" failed. No retries permitted until 2026-08-01 01:15:22.856330134 +0000 UTC m=+2489.090262378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls") pod "maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf" (UID: "6d62a22a-31b0-498c-b5e1-15886a2df122") : secret "maas-api-e2e-mt-ff21edf2-serving-cert" not found
Aug 01 01:15:22.356923 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.356905    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d62a22a-31b0-498c-b5e1-15886a2df122-openshift-service-ca\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.365004 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.364984    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmsc\" (UniqueName: \"kubernetes.io/projected/6d62a22a-31b0-498c-b5e1-15886a2df122-kube-api-access-rnmsc\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.770966 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.770915    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"]
Aug 01 01:15:22.774466 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.774446    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:22.777025 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.776995    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ff21edf2-dockercfg-25m55\""
Aug 01 01:15:22.778820 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.778801    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ff21edf2\""
Aug 01 01:15:22.784154 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.784128    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"]
Aug 01 01:15:22.861945 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.861901    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.862171 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.861979    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwrgr\" (UniqueName: \"kubernetes.io/projected/e6f4efab-d9b2-4796-9e8d-2e97e992050d-kube-api-access-zwrgr\") pod \"payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw\" (UID: \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:22.862171 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.862042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4efab-d9b2-4796-9e8d-2e97e992050d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw\" (UID: \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:22.864730 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.864698    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls\") pod \"maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:22.963380 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.963327    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4efab-d9b2-4796-9e8d-2e97e992050d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw\" (UID: \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:22.963565 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.963473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwrgr\" (UniqueName: \"kubernetes.io/projected/e6f4efab-d9b2-4796-9e8d-2e97e992050d-kube-api-access-zwrgr\") pod \"payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw\" (UID: \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:22.964111 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.964087    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4efab-d9b2-4796-9e8d-2e97e992050d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw\" (UID: \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:22.971628 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:22.971595    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwrgr\" (UniqueName: \"kubernetes.io/projected/e6f4efab-d9b2-4796-9e8d-2e97e992050d-kube-api-access-zwrgr\") pod \"payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw\" (UID: \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:23.083335 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:23.083304    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:23.086045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:23.086005    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:23.229443 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:23.229409    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"]
Aug 01 01:15:23.232094 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:15:23.232064    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d62a22a_31b0_498c_b5e1_15886a2df122.slice/crio-219b6202fb3e50ff16e8cec95aaa00f88c92ed27609a726e5a93138636c01878 WatchSource:0}: Error finding container 219b6202fb3e50ff16e8cec95aaa00f88c92ed27609a726e5a93138636c01878: Status 404 returned error can't find the container with id 219b6202fb3e50ff16e8cec95aaa00f88c92ed27609a726e5a93138636c01878
Aug 01 01:15:23.233908 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:23.233847    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 01:15:23.248875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:23.248848    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"]
Aug 01 01:15:23.250741 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:15:23.250718    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f4efab_d9b2_4796_9e8d_2e97e992050d.slice/crio-c5b60ba16a6208e505556315d23c5f3a5682e4cffc0448430731f7fe2f3c1d95 WatchSource:0}: Error finding container c5b60ba16a6208e505556315d23c5f3a5682e4cffc0448430731f7fe2f3c1d95: Status 404 returned error can't find the container with id c5b60ba16a6208e505556315d23c5f3a5682e4cffc0448430731f7fe2f3c1d95
Aug 01 01:15:24.153424 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:24.153392    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf" event={"ID":"6d62a22a-31b0-498c-b5e1-15886a2df122","Type":"ContainerStarted","Data":"c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4"}
Aug 01 01:15:24.153662 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:24.153446    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf" event={"ID":"6d62a22a-31b0-498c-b5e1-15886a2df122","Type":"ContainerStarted","Data":"219b6202fb3e50ff16e8cec95aaa00f88c92ed27609a726e5a93138636c01878"}
Aug 01 01:15:24.153662 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:24.153489    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:24.155222 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:24.155191    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw" event={"ID":"e6f4efab-d9b2-4796-9e8d-2e97e992050d","Type":"ContainerStarted","Data":"34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa"}
Aug 01 01:15:24.155348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:24.155254    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:24.155348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:24.155279    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw" event={"ID":"e6f4efab-d9b2-4796-9e8d-2e97e992050d","Type":"ContainerStarted","Data":"c5b60ba16a6208e505556315d23c5f3a5682e4cffc0448430731f7fe2f3c1d95"}
Aug 01 01:15:24.170378 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:24.170315    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf" podStartSLOduration=1.6707448280000001 podStartE2EDuration="2.170296448s" totalImagesPullingTime="499.55162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:15:24.16809464 +0000 UTC m=+2490.402026907" watchObservedRunningTime="2026-08-01 01:15:24.170296448 +0000 UTC m=+2490.404228715"
Aug 01 01:15:24.183019 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:24.182974    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw" podStartSLOduration=2.182959971 podStartE2EDuration="2.182959971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:15:24.181495053 +0000 UTC m=+2490.415427321" watchObservedRunningTime="2026-08-01 01:15:24.182959971 +0000 UTC m=+2490.416892237"
Aug 01 01:15:30.169130 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:30.169090    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:15:35.131061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.131024    2570 patch_prober.go:28] interesting pod/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Aug 01 01:15:35.131472 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.131095    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 01:15:35.162249 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.162221    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:15:35.217999 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.217961    2570 generic.go:356] "Generic (PLEG): container finished" podID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerID="9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0" exitCode=0
Aug 01 01:15:35.218196 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.218038    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" event={"ID":"45884bcb-c6b4-47e9-acd4-166c5e45bf91","Type":"ContainerDied","Data":"9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0"}
Aug 01 01:15:35.218608 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.218590    2570 scope.go:117] "RemoveContainer" containerID="9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0"
Aug 01 01:15:35.220034 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.220003    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:15:35.220136 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.220076    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:15:35.220136 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:35.220107    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:15:36.224787 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:36.224747    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" event={"ID":"45884bcb-c6b4-47e9-acd4-166c5e45bf91","Type":"ContainerStarted","Data":"9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba"}
Aug 01 01:15:37.088197 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:37.088158    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:37.088486 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:37.088460    2570 patch_prober.go:28] interesting pod/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Aug 01 01:15:37.088567 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:37.088548    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 01:15:38.088599 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:38.088558    2570 patch_prober.go:28] interesting pod/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Aug 01 01:15:38.089069 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:38.088618    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 01:15:39.087639 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:39.087600    2570 patch_prober.go:28] interesting pod/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Aug 01 01:15:39.087833 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:39.087659    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 01:15:40.088135 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:40.088095    2570 patch_prober.go:28] interesting pod/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Aug 01 01:15:40.088551 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:40.088158    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 01:15:41.087949 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:41.087900    2570 patch_prober.go:28] interesting pod/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Aug 01 01:15:41.088172 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:41.088011    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 01:15:42.092005 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:42.091971    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:42.092503 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:42.092354    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:42.093031 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:42.093008    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:15:54.493291 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:15:54.493218    2570 scope.go:117] "RemoveContainer" containerID="b970ec45ba0f1d72ec4d999f4bd8222f02a75f591b8b6e1a348a88d96c322b2e"
Aug 01 01:16:00.694412 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.694375    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"]
Aug 01 01:16:00.697921 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.697904    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.703439 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.703380    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ff21edf2\"/\"kube-root-ca.crt\""
Aug 01 01:16:00.703581 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.703517    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ff21edf2\"/\"openshift-service-ca.crt\""
Aug 01 01:16:00.704648 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.704624    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ff21edf2\"/\"e2e-lifecycle-model-ff21edf2-kserve-self-signed-certs\""
Aug 01 01:16:00.705095 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.705069    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ff21edf2\"/\"default-dockercfg-pdswx\""
Aug 01 01:16:00.707559 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.707532    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"]
Aug 01 01:16:00.801587 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.801551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9dwn\" (UniqueName: \"kubernetes.io/projected/b043e21c-0f62-479a-8a4e-696141fd2938-kube-api-access-d9dwn\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.801754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.801593    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-tmp-dir\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.801754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.801664    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b043e21c-0f62-479a-8a4e-696141fd2938-tls-certs\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.801754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.801747    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-home\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.801867 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.801774    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-dshm\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.801867 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.801796    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-model-cache\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.902889 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.902850    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9dwn\" (UniqueName: \"kubernetes.io/projected/b043e21c-0f62-479a-8a4e-696141fd2938-kube-api-access-d9dwn\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.903106 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.902898    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-tmp-dir\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.903106 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.902925    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b043e21c-0f62-479a-8a4e-696141fd2938-tls-certs\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.903106 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.902980    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-home\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.903106 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.903007    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-dshm\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.903106 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.903039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-model-cache\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.903458 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.903421    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-tmp-dir\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.903458 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.903439    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-model-cache\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.903601 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.903514    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-home\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.905602 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.905580    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-dshm\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.905862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.905839    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b043e21c-0f62-479a-8a4e-696141fd2938-tls-certs\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:00.911149 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:00.911127    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9dwn\" (UniqueName: \"kubernetes.io/projected/b043e21c-0f62-479a-8a4e-696141fd2938-kube-api-access-d9dwn\") pod \"e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") " pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:01.008017 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:01.007904    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:01.142246 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:01.142219    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"]
Aug 01 01:16:01.144398 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:16:01.144370    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb043e21c_0f62_479a_8a4e_696141fd2938.slice/crio-2187516d3bb6f6a609b9526bca8ab3024842febe4eb4ae87461555f2097c8e52 WatchSource:0}: Error finding container 2187516d3bb6f6a609b9526bca8ab3024842febe4eb4ae87461555f2097c8e52: Status 404 returned error can't find the container with id 2187516d3bb6f6a609b9526bca8ab3024842febe4eb4ae87461555f2097c8e52
Aug 01 01:16:01.344973 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:01.344908    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d" event={"ID":"b043e21c-0f62-479a-8a4e-696141fd2938","Type":"ContainerStarted","Data":"d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56"}
Aug 01 01:16:01.345123 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:01.344977    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d" event={"ID":"b043e21c-0f62-479a-8a4e-696141fd2938","Type":"ContainerStarted","Data":"2187516d3bb6f6a609b9526bca8ab3024842febe4eb4ae87461555f2097c8e52"}
Aug 01 01:16:01.361153 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:01.361099    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d" podStartSLOduration=1.3610842060000001 podStartE2EDuration="1.361084206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:16:01.3601102 +0000 UTC m=+2527.594042466" watchObservedRunningTime="2026-08-01 01:16:01.361084206 +0000 UTC m=+2527.595016472"
Aug 01 01:16:11.008111 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:11.008074    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:11.008584 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:11.008134    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:11.020697 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:11.020670    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:11.400261 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:11.400228    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:16:41.196564 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.196528    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"]
Aug 01 01:16:41.197148 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.196765    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf" podUID="6d62a22a-31b0-498c-b5e1-15886a2df122" containerName="maas-api" containerID="cri-o://c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4" gracePeriod=30
Aug 01 01:16:41.253499 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.253466    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"]
Aug 01 01:16:41.253906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.253877    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw" podUID="e6f4efab-d9b2-4796-9e8d-2e97e992050d" containerName="payload-pre-processing" containerID="cri-o://34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa" gracePeriod=30
Aug 01 01:16:41.505190 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.505163    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:16:41.533951 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.533899    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:16:41.538667 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.538642    2570 generic.go:356] "Generic (PLEG): container finished" podID="6d62a22a-31b0-498c-b5e1-15886a2df122" containerID="c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4" exitCode=0
Aug 01 01:16:41.538807 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.538697    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"
Aug 01 01:16:41.538807 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.538731    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf" event={"ID":"6d62a22a-31b0-498c-b5e1-15886a2df122","Type":"ContainerDied","Data":"c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4"}
Aug 01 01:16:41.538958 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.538817    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf" event={"ID":"6d62a22a-31b0-498c-b5e1-15886a2df122","Type":"ContainerDied","Data":"219b6202fb3e50ff16e8cec95aaa00f88c92ed27609a726e5a93138636c01878"}
Aug 01 01:16:41.538958 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.538853    2570 scope.go:117] "RemoveContainer" containerID="c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4"
Aug 01 01:16:41.540192 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.540169    2570 generic.go:356] "Generic (PLEG): container finished" podID="e6f4efab-d9b2-4796-9e8d-2e97e992050d" containerID="34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa" exitCode=0
Aug 01 01:16:41.540321 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.540203    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"
Aug 01 01:16:41.540321 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.540203    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw" event={"ID":"e6f4efab-d9b2-4796-9e8d-2e97e992050d","Type":"ContainerDied","Data":"34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa"}
Aug 01 01:16:41.540321 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.540241    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw" event={"ID":"e6f4efab-d9b2-4796-9e8d-2e97e992050d","Type":"ContainerDied","Data":"c5b60ba16a6208e505556315d23c5f3a5682e4cffc0448430731f7fe2f3c1d95"}
Aug 01 01:16:41.547083 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.547061    2570 scope.go:117] "RemoveContainer" containerID="c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4"
Aug 01 01:16:41.547353 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:16:41.547334    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4\": container with ID starting with c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4 not found: ID does not exist" containerID="c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4"
Aug 01 01:16:41.547413 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.547362    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4"} err="failed to get container status \"c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4\": rpc error: code = NotFound desc = could not find container \"c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4\": container with ID starting with c47d97cf61c50ff2ec4d66c6557110cffebc9523b0460de42d89025b6f973df4 not found: ID does not exist"
Aug 01 01:16:41.547413 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.547378    2570 scope.go:117] "RemoveContainer" containerID="34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa"
Aug 01 01:16:41.556999 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.556982    2570 scope.go:117] "RemoveContainer" containerID="34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa"
Aug 01 01:16:41.557266 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:16:41.557246    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa\": container with ID starting with 34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa not found: ID does not exist" containerID="34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa"
Aug 01 01:16:41.557332 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.557273    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa"} err="failed to get container status \"34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa\": rpc error: code = NotFound desc = could not find container \"34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa\": container with ID starting with 34523b7616e899b04c1685a898490693c7942375e862d8fb965781f5434930aa not found: ID does not exist"
Aug 01 01:16:41.657041 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.657005    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d62a22a-31b0-498c-b5e1-15886a2df122-openshift-service-ca\") pod \"6d62a22a-31b0-498c-b5e1-15886a2df122\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") "
Aug 01 01:16:41.657041 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.657045    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwrgr\" (UniqueName: \"kubernetes.io/projected/e6f4efab-d9b2-4796-9e8d-2e97e992050d-kube-api-access-zwrgr\") pod \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\" (UID: \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\") "
Aug 01 01:16:41.657233 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.657090    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4efab-d9b2-4796-9e8d-2e97e992050d-plugins-config-volume\") pod \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\" (UID: \"e6f4efab-d9b2-4796-9e8d-2e97e992050d\") "
Aug 01 01:16:41.657233 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.657124    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls\") pod \"6d62a22a-31b0-498c-b5e1-15886a2df122\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") "
Aug 01 01:16:41.657233 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.657149    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnmsc\" (UniqueName: \"kubernetes.io/projected/6d62a22a-31b0-498c-b5e1-15886a2df122-kube-api-access-rnmsc\") pod \"6d62a22a-31b0-498c-b5e1-15886a2df122\" (UID: \"6d62a22a-31b0-498c-b5e1-15886a2df122\") "
Aug 01 01:16:41.657395 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.657357    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d62a22a-31b0-498c-b5e1-15886a2df122-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d62a22a-31b0-498c-b5e1-15886a2df122" (UID: "6d62a22a-31b0-498c-b5e1-15886a2df122"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:16:41.657456 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.657433    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d62a22a-31b0-498c-b5e1-15886a2df122-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:16:41.657541 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.657516    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f4efab-d9b2-4796-9e8d-2e97e992050d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6f4efab-d9b2-4796-9e8d-2e97e992050d" (UID: "e6f4efab-d9b2-4796-9e8d-2e97e992050d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:16:41.659259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.659233    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f4efab-d9b2-4796-9e8d-2e97e992050d-kube-api-access-zwrgr" (OuterVolumeSpecName: "kube-api-access-zwrgr") pod "e6f4efab-d9b2-4796-9e8d-2e97e992050d" (UID: "e6f4efab-d9b2-4796-9e8d-2e97e992050d"). InnerVolumeSpecName "kube-api-access-zwrgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:16:41.659348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.659289    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d62a22a-31b0-498c-b5e1-15886a2df122" (UID: "6d62a22a-31b0-498c-b5e1-15886a2df122"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:16:41.659348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.659299    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d62a22a-31b0-498c-b5e1-15886a2df122-kube-api-access-rnmsc" (OuterVolumeSpecName: "kube-api-access-rnmsc") pod "6d62a22a-31b0-498c-b5e1-15886a2df122" (UID: "6d62a22a-31b0-498c-b5e1-15886a2df122"). InnerVolumeSpecName "kube-api-access-rnmsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:16:41.758167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.758071    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4efab-d9b2-4796-9e8d-2e97e992050d-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:16:41.758167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.758109    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d62a22a-31b0-498c-b5e1-15886a2df122-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:16:41.758167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.758119    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmsc\" (UniqueName: \"kubernetes.io/projected/6d62a22a-31b0-498c-b5e1-15886a2df122-kube-api-access-rnmsc\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:16:41.758167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.758127    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwrgr\" (UniqueName: \"kubernetes.io/projected/e6f4efab-d9b2-4796-9e8d-2e97e992050d-kube-api-access-zwrgr\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:16:41.864524 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.864488    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"]
Aug 01 01:16:41.866234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.866199    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff21edf2-68dc5dfc4b-x8rhf"]
Aug 01 01:16:41.876911 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.876879    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"]
Aug 01 01:16:41.878651 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:41.878630    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff21edf2-767f5f77d4-rlkzw"]
Aug 01 01:16:42.261923 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:42.261890    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d62a22a-31b0-498c-b5e1-15886a2df122" path="/var/lib/kubelet/pods/6d62a22a-31b0-498c-b5e1-15886a2df122/volumes"
Aug 01 01:16:42.262348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:42.262251    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6f4efab-d9b2-4796-9e8d-2e97e992050d" path="/var/lib/kubelet/pods/e6f4efab-d9b2-4796-9e8d-2e97e992050d/volumes"
Aug 01 01:16:49.625322 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:49.625282    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"]
Aug 01 01:16:54.511972 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:16:54.511943    2570 scope.go:117] "RemoveContainer" containerID="5df24d4f90120e6ab6b03a727c449d68c44186dd693306b8c6ef34f0e9db9ce1"
Aug 01 01:17:04.644186 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.644149    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d" podUID="b043e21c-0f62-479a-8a4e-696141fd2938" containerName="main" containerID="cri-o://d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56" gracePeriod=45
Aug 01 01:17:04.900120 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.900051    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:17:04.959645 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.959615    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-tmp-dir\") pod \"b043e21c-0f62-479a-8a4e-696141fd2938\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") "
Aug 01 01:17:04.959819 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.959656    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-model-cache\") pod \"b043e21c-0f62-479a-8a4e-696141fd2938\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") "
Aug 01 01:17:04.959819 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.959710    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b043e21c-0f62-479a-8a4e-696141fd2938-tls-certs\") pod \"b043e21c-0f62-479a-8a4e-696141fd2938\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") "
Aug 01 01:17:04.959819 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.959749    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-home\") pod \"b043e21c-0f62-479a-8a4e-696141fd2938\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") "
Aug 01 01:17:04.959819 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.959805    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9dwn\" (UniqueName: \"kubernetes.io/projected/b043e21c-0f62-479a-8a4e-696141fd2938-kube-api-access-d9dwn\") pod \"b043e21c-0f62-479a-8a4e-696141fd2938\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") "
Aug 01 01:17:04.960116 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.959830    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-dshm\") pod \"b043e21c-0f62-479a-8a4e-696141fd2938\" (UID: \"b043e21c-0f62-479a-8a4e-696141fd2938\") "
Aug 01 01:17:04.960116 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.959954    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-model-cache" (OuterVolumeSpecName: "model-cache") pod "b043e21c-0f62-479a-8a4e-696141fd2938" (UID: "b043e21c-0f62-479a-8a4e-696141fd2938"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:04.960116 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.959951    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b043e21c-0f62-479a-8a4e-696141fd2938" (UID: "b043e21c-0f62-479a-8a4e-696141fd2938"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:04.960277 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.960142    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:04.960277 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.960149    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-home" (OuterVolumeSpecName: "home") pod "b043e21c-0f62-479a-8a4e-696141fd2938" (UID: "b043e21c-0f62-479a-8a4e-696141fd2938"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:04.960277 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.960164    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:04.962079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.962054    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-dshm" (OuterVolumeSpecName: "dshm") pod "b043e21c-0f62-479a-8a4e-696141fd2938" (UID: "b043e21c-0f62-479a-8a4e-696141fd2938"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:04.962201 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.962148    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b043e21c-0f62-479a-8a4e-696141fd2938-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b043e21c-0f62-479a-8a4e-696141fd2938" (UID: "b043e21c-0f62-479a-8a4e-696141fd2938"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:17:04.962272 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:04.962252    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b043e21c-0f62-479a-8a4e-696141fd2938-kube-api-access-d9dwn" (OuterVolumeSpecName: "kube-api-access-d9dwn") pod "b043e21c-0f62-479a-8a4e-696141fd2938" (UID: "b043e21c-0f62-479a-8a4e-696141fd2938"). InnerVolumeSpecName "kube-api-access-d9dwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:17:05.061092 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.061044    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9dwn\" (UniqueName: \"kubernetes.io/projected/b043e21c-0f62-479a-8a4e-696141fd2938-kube-api-access-d9dwn\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:05.061092 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.061086    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:05.061092 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.061097    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b043e21c-0f62-479a-8a4e-696141fd2938-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:05.061092 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.061107    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b043e21c-0f62-479a-8a4e-696141fd2938-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:05.658050 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.658017    2570 generic.go:356] "Generic (PLEG): container finished" podID="b043e21c-0f62-479a-8a4e-696141fd2938" containerID="d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56" exitCode=0
Aug 01 01:17:05.658550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.658059    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"
Aug 01 01:17:05.658550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.658107    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d" event={"ID":"b043e21c-0f62-479a-8a4e-696141fd2938","Type":"ContainerDied","Data":"d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56"}
Aug 01 01:17:05.658550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.658156    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d" event={"ID":"b043e21c-0f62-479a-8a4e-696141fd2938","Type":"ContainerDied","Data":"2187516d3bb6f6a609b9526bca8ab3024842febe4eb4ae87461555f2097c8e52"}
Aug 01 01:17:05.658550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.658180    2570 scope.go:117] "RemoveContainer" containerID="d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56"
Aug 01 01:17:05.666673 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.666655    2570 scope.go:117] "RemoveContainer" containerID="d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56"
Aug 01 01:17:05.666982 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:17:05.666964    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56\": container with ID starting with d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56 not found: ID does not exist" containerID="d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56"
Aug 01 01:17:05.667064 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.666988    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56"} err="failed to get container status \"d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56\": rpc error: code = NotFound desc = could not find container \"d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56\": container with ID starting with d9b4220225944d1de22544e7561c34838ae331d7dda02b89d3a2902d6b33ca56 not found: ID does not exist"
Aug 01 01:17:05.677979 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.677955    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"]
Aug 01 01:17:05.683468 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:05.683433    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ff21edf2/e2e-lifecycle-model-ff21edf2-kserve-cd9c96f55-lxr8d"]
Aug 01 01:17:06.261591 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:06.261556    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b043e21c-0f62-479a-8a4e-696141fd2938" path="/var/lib/kubelet/pods/b043e21c-0f62-479a-8a4e-696141fd2938/volumes"
Aug 01 01:17:32.496351 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:32.496310    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"]
Aug 01 01:17:32.496884 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:32.496565    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy" containerID="cri-o://9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba" gracePeriod=30
Aug 01 01:17:37.744318 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.744289    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:17:37.806454 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.806377    2570 generic.go:356] "Generic (PLEG): container finished" podID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerID="9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba" exitCode=0
Aug 01 01:17:37.806454 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.806429    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"
Aug 01 01:17:37.806630 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.806464    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" event={"ID":"45884bcb-c6b4-47e9-acd4-166c5e45bf91","Type":"ContainerDied","Data":"9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba"}
Aug 01 01:17:37.806630 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.806522    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8" event={"ID":"45884bcb-c6b4-47e9-acd4-166c5e45bf91","Type":"ContainerDied","Data":"69fe477c8fa4d57373732e253a4aeae93051cd69356e89956579403c58441f84"}
Aug 01 01:17:37.806630 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.806552    2570 scope.go:117] "RemoveContainer" containerID="9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba"
Aug 01 01:17:37.814880 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.814853    2570 scope.go:117] "RemoveContainer" containerID="9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0"
Aug 01 01:17:37.822316 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.822299    2570 scope.go:117] "RemoveContainer" containerID="9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba"
Aug 01 01:17:37.822556 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:17:37.822537    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba\": container with ID starting with 9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba not found: ID does not exist" containerID="9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba"
Aug 01 01:17:37.822607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.822565    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba"} err="failed to get container status \"9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba\": rpc error: code = NotFound desc = could not find container \"9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba\": container with ID starting with 9d3aaba8cd670ec95719125796f2a4cf50998035c9f643b999ab613a6bba25ba not found: ID does not exist"
Aug 01 01:17:37.822607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.822582    2570 scope.go:117] "RemoveContainer" containerID="9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0"
Aug 01 01:17:37.822813 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:17:37.822797    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0\": container with ID starting with 9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0 not found: ID does not exist" containerID="9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0"
Aug 01 01:17:37.822858 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.822816    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0"} err="failed to get container status \"9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0\": rpc error: code = NotFound desc = could not find container \"9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0\": container with ID starting with 9d78cfad38a2b44057342783de9b35157bd080f6b9229cc9595c41ea453ffbf0 not found: ID does not exist"
Aug 01 01:17:37.845985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.845967    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-envoy\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846097 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.845998    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-credential-socket\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846097 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846030    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istiod-ca-cert\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846097 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846054    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-socket\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846097 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846083    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-data\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846130    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-podinfo\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846156    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2dk5\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-kube-api-access-m2dk5\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846204    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-token\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846264    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:37.846306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846276    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-certs\") pod \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\" (UID: \"45884bcb-c6b4-47e9-acd4-166c5e45bf91\") "
Aug 01 01:17:37.846558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846362    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-data" (OuterVolumeSpecName: "istio-data") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:37.846558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846372    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:37.846558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846515    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:37.846558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846526    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:17:37.846763 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846641    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:37.846763 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846658    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:37.846763 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846672    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:37.846763 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846685    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:37.846763 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.846699    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:37.848030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.848002    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:17:37.848299 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.848274    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-kube-api-access-m2dk5" (OuterVolumeSpecName: "kube-api-access-m2dk5") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "kube-api-access-m2dk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:17:37.848476 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.848457    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-token" (OuterVolumeSpecName: "istio-token") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:17:37.848565 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.848482    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45884bcb-c6b4-47e9-acd4-166c5e45bf91" (UID: "45884bcb-c6b4-47e9-acd4-166c5e45bf91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:17:37.948053 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.948024    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:37.948053 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.948048    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:37.948053 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.948057    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45884bcb-c6b4-47e9-acd4-166c5e45bf91-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:37.948251 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:37.948068    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2dk5\" (UniqueName: \"kubernetes.io/projected/45884bcb-c6b4-47e9-acd4-166c5e45bf91-kube-api-access-m2dk5\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:17:38.128645 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:38.128615    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"]
Aug 01 01:17:38.133763 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:38.133741    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ff21edf2-openshift-default-7b5b759cb8-28mn8"]
Aug 01 01:17:38.261417 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:17:38.261389    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" path="/var/lib/kubelet/pods/45884bcb-c6b4-47e9-acd4-166c5e45bf91/volumes"
Aug 01 01:18:38.088166 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088134    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"]
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088469    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b043e21c-0f62-479a-8a4e-696141fd2938" containerName="main"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088486    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b043e21c-0f62-479a-8a4e-696141fd2938" containerName="main"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088500    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d62a22a-31b0-498c-b5e1-15886a2df122" containerName="maas-api"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088525    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d62a22a-31b0-498c-b5e1-15886a2df122" containerName="maas-api"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088534    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088541    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088552    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f4efab-d9b2-4796-9e8d-2e97e992050d" containerName="payload-pre-processing"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088557    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f4efab-d9b2-4796-9e8d-2e97e992050d" containerName="payload-pre-processing"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088568    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088574    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088633    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f4efab-d9b2-4796-9e8d-2e97e992050d" containerName="payload-pre-processing"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088643    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088648    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b043e21c-0f62-479a-8a4e-696141fd2938" containerName="main"
Aug 01 01:18:38.088647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.088654    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d62a22a-31b0-498c-b5e1-15886a2df122" containerName="maas-api"
Aug 01 01:18:38.091161 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.091144    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.093914 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.093889    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Aug 01 01:18:38.093914 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.093913    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f6a6f5-openshift-default-dockercfg-6czjm\""
Aug 01 01:18:38.107236 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.107208    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"]
Aug 01 01:18:38.239965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.239913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-certs\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.240142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.239990    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-podinfo\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.240142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.240033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-socket\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.240142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.240063    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-envoy\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.240142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.240093    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istiod-ca-cert\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.240142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.240127    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-data\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.240372 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.240165    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-credential-socket\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.240372 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.240193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-token\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.240372 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.240219    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgbl\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-kube-api-access-csgbl\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341466 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-certs\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341466 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341430    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-podinfo\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341466 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341451    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-socket\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341674 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-envoy\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341674 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341625    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istiod-ca-cert\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341674 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341659    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-data\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341832 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341702    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-credential-socket\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341832 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341820    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-certs\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341973 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341851    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-socket\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341973 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341864    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-token\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.341973 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341906    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgbl\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-kube-api-access-csgbl\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.342155 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.341989    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-credential-socket\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.342155 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.342085    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-data\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.342430 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.342407    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istiod-ca-cert\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.343869 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.343836    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-envoy\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.344009 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.343992    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-podinfo\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.349293 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.349269    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgbl\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-kube-api-access-csgbl\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.349409 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.349300    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-token\") pod \"e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") " pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.402093 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.402068    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:38.542559 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.542523    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"]
Aug 01 01:18:38.544957 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:18:38.544916    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ca871b_2b99_4ec3_9cdb_a9c86b8e3ed0.slice/crio-ef0a326396975ec7e83d8625a2f225a2c3ae71614b076fd78b07b55b144b9b16 WatchSource:0}: Error finding container ef0a326396975ec7e83d8625a2f225a2c3ae71614b076fd78b07b55b144b9b16: Status 404 returned error can't find the container with id ef0a326396975ec7e83d8625a2f225a2c3ae71614b076fd78b07b55b144b9b16
Aug 01 01:18:38.546978 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.546945    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:18:38.547044 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.547014    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:18:38.547084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:38.547070    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:18:39.090864 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:39.090826    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" event={"ID":"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0","Type":"ContainerStarted","Data":"1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e"}
Aug 01 01:18:39.091273 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:39.090877    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" event={"ID":"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0","Type":"ContainerStarted","Data":"ef0a326396975ec7e83d8625a2f225a2c3ae71614b076fd78b07b55b144b9b16"}
Aug 01 01:18:39.111632 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:39.111578    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podStartSLOduration=1.111558071 podStartE2EDuration="1.111558071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:18:39.109522628 +0000 UTC m=+2685.343454895" watchObservedRunningTime="2026-08-01 01:18:39.111558071 +0000 UTC m=+2685.345490337"
Aug 01 01:18:39.402443 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:39.402376    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:39.406804 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:39.406782    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:40.096102 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:40.096070    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:40.097029 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:40.097010    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:42.147745 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.147700    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"]
Aug 01 01:18:42.148354 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.148333    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="45884bcb-c6b4-47e9-acd4-166c5e45bf91" containerName="istio-proxy"
Aug 01 01:18:42.151266 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.151247    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.153882 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.153858    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f6a6f5-serving-cert\""
Aug 01 01:18:42.160988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.160967    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"]
Aug 01 01:18:42.272910 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.272882    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gntz\" (UniqueName: \"kubernetes.io/projected/45251bd8-26de-4a44-953a-f5e9b4f0e629-kube-api-access-7gntz\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.273100 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.273031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45251bd8-26de-4a44-953a-f5e9b4f0e629-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.273100 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.273079    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.373476 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.373445    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.373641 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.373489    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gntz\" (UniqueName: \"kubernetes.io/projected/45251bd8-26de-4a44-953a-f5e9b4f0e629-kube-api-access-7gntz\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.373641 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.373553    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45251bd8-26de-4a44-953a-f5e9b4f0e629-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.373641 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:18:42.373583    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-serving-cert: secret "maas-api-e2e-inf-a-f6a6f5-serving-cert" not found
Aug 01 01:18:42.373750 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:18:42.373652    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls podName:45251bd8-26de-4a44-953a-f5e9b4f0e629 nodeName:}" failed. No retries permitted until 2026-08-01 01:18:42.873635452 +0000 UTC m=+2689.107567695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls") pod "maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc" (UID: "45251bd8-26de-4a44-953a-f5e9b4f0e629") : secret "maas-api-e2e-inf-a-f6a6f5-serving-cert" not found
Aug 01 01:18:42.374219 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.374201    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45251bd8-26de-4a44-953a-f5e9b4f0e629-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.391813 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.391791    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gntz\" (UniqueName: \"kubernetes.io/projected/45251bd8-26de-4a44-953a-f5e9b4f0e629-kube-api-access-7gntz\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.858814 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.858777    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"]
Aug 01 01:18:42.862190 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.862173    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:42.865061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.865036    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f6a6f5-dockercfg-j5s57\""
Aug 01 01:18:42.865200 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.865121    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f6a6f5\""
Aug 01 01:18:42.875280 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.875255    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"]
Aug 01 01:18:42.877422 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.877402    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.880144 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.880119    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls\") pod \"maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:42.978207 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.978171    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be06045-5456-43de-be0b-a37afb466886-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x\" (UID: \"6be06045-5456-43de-be0b-a37afb466886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:42.978207 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:42.978208    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8jt\" (UniqueName: \"kubernetes.io/projected/6be06045-5456-43de-be0b-a37afb466886-kube-api-access-rb8jt\") pod \"payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x\" (UID: \"6be06045-5456-43de-be0b-a37afb466886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:43.064018 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:43.063988    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:43.079342 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:43.079310    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be06045-5456-43de-be0b-a37afb466886-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x\" (UID: \"6be06045-5456-43de-be0b-a37afb466886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:43.079463 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:43.079357    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8jt\" (UniqueName: \"kubernetes.io/projected/6be06045-5456-43de-be0b-a37afb466886-kube-api-access-rb8jt\") pod \"payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x\" (UID: \"6be06045-5456-43de-be0b-a37afb466886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:43.080240 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:43.080215    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be06045-5456-43de-be0b-a37afb466886-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x\" (UID: \"6be06045-5456-43de-be0b-a37afb466886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:43.087188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:43.087168    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8jt\" (UniqueName: \"kubernetes.io/projected/6be06045-5456-43de-be0b-a37afb466886-kube-api-access-rb8jt\") pod \"payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x\" (UID: \"6be06045-5456-43de-be0b-a37afb466886\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:43.172417 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:43.172351    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:43.209255 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:43.209216    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"]
Aug 01 01:18:43.210116 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:18:43.210084    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45251bd8_26de_4a44_953a_f5e9b4f0e629.slice/crio-8cfa4a6efaa9126f8f06fc81b081c150756eb54b5b1fdfb0f0080d981c8835ee WatchSource:0}: Error finding container 8cfa4a6efaa9126f8f06fc81b081c150756eb54b5b1fdfb0f0080d981c8835ee: Status 404 returned error can't find the container with id 8cfa4a6efaa9126f8f06fc81b081c150756eb54b5b1fdfb0f0080d981c8835ee
Aug 01 01:18:43.307924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:43.307898    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"]
Aug 01 01:18:43.310174 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:18:43.310150    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be06045_5456_43de_be0b_a37afb466886.slice/crio-bd18a72eabbe8d1869fdc08d44e8a049aaf9737182ea747df8dc00baa9aedeca WatchSource:0}: Error finding container bd18a72eabbe8d1869fdc08d44e8a049aaf9737182ea747df8dc00baa9aedeca: Status 404 returned error can't find the container with id bd18a72eabbe8d1869fdc08d44e8a049aaf9737182ea747df8dc00baa9aedeca
Aug 01 01:18:44.116154 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:44.116113    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc" event={"ID":"45251bd8-26de-4a44-953a-f5e9b4f0e629","Type":"ContainerStarted","Data":"ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193"}
Aug 01 01:18:44.116372 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:44.116181    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:44.116372 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:44.116209    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc" event={"ID":"45251bd8-26de-4a44-953a-f5e9b4f0e629","Type":"ContainerStarted","Data":"8cfa4a6efaa9126f8f06fc81b081c150756eb54b5b1fdfb0f0080d981c8835ee"}
Aug 01 01:18:44.117646 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:44.117617    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x" event={"ID":"6be06045-5456-43de-be0b-a37afb466886","Type":"ContainerStarted","Data":"55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5"}
Aug 01 01:18:44.117757 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:44.117665    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x" event={"ID":"6be06045-5456-43de-be0b-a37afb466886","Type":"ContainerStarted","Data":"bd18a72eabbe8d1869fdc08d44e8a049aaf9737182ea747df8dc00baa9aedeca"}
Aug 01 01:18:44.117757 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:44.117745    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:44.131016 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:44.130966    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc" podStartSLOduration=1.721324026 podStartE2EDuration="2.130925588s" totalImagesPullingTime="409.601562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:18:44.13032018 +0000 UTC m=+2690.364252445" watchObservedRunningTime="2026-08-01 01:18:44.130925588 +0000 UTC m=+2690.364857856"
Aug 01 01:18:44.145227 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:44.145188    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x" podStartSLOduration=2.145175203 podStartE2EDuration="2.145175203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:18:44.144097446 +0000 UTC m=+2690.378029713" watchObservedRunningTime="2026-08-01 01:18:44.145175203 +0000 UTC m=+2690.379107470"
Aug 01 01:18:50.128334 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:50.128250    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:18:55.097508 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.097419    2570 patch_prober.go:28] interesting pod/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Aug 01 01:18:55.097508 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.097483    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Aug 01 01:18:55.124213 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.124182    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:18:55.175075 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.174993    2570 generic.go:356] "Generic (PLEG): container finished" podID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerID="1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e" exitCode=0
Aug 01 01:18:55.175075 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.175032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" event={"ID":"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0","Type":"ContainerDied","Data":"1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e"}
Aug 01 01:18:55.175468 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.175450    2570 scope.go:117] "RemoveContainer" containerID="1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e"
Aug 01 01:18:55.176509 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.176482    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:18:55.176600 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.176545    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:18:55.176600 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:55.176576    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:18:56.181649 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:56.181611    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" event={"ID":"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0","Type":"ContainerStarted","Data":"98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668"}
Aug 01 01:18:56.402492 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:56.402454    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:18:56.402710 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:56.402685    2570 patch_prober.go:28] interesting pod/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Aug 01 01:18:56.402763 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:56.402749    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Aug 01 01:18:57.402462 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:57.402427    2570 patch_prober.go:28] interesting pod/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Aug 01 01:18:57.402878 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:57.402483    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Aug 01 01:18:58.403361 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:58.403323    2570 patch_prober.go:28] interesting pod/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Aug 01 01:18:58.403778 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:58.403378    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Aug 01 01:18:59.403186 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:59.403150    2570 patch_prober.go:28] interesting pod/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Aug 01 01:18:59.403390 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:18:59.403203    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Aug 01 01:19:00.402652 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:00.402610    2570 patch_prober.go:28] interesting pod/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Aug 01 01:19:00.402873 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:00.402699    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Aug 01 01:19:01.407398 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:01.407362    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:19:01.407835 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:01.407633    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:19:01.408190 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:01.408175    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:19:06.827686 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.827649    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"]
Aug 01 01:19:06.831008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.830988    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.833739 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.833718    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1cf076-openshift-default-dockercfg-zcrg7\""
Aug 01 01:19:06.841993 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.841971    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"]
Aug 01 01:19:06.897500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897467    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istiod-ca-cert\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.897500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897503    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-data\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.897721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897521    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-podinfo\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.897721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897545    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-certs\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.897721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897620    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-token\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.897721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897676    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-envoy\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.897721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897714    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-credential-socket\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.897967 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897749    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-socket\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.897967 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.897779    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgf9\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-kube-api-access-sxgf9\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.998477 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998440    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istiod-ca-cert\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.998661 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998484    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-data\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.998661 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998503    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-podinfo\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.998661 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998630    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-certs\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.998845 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998692    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-token\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.998845 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-envoy\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.998845 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998787    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-credential-socket\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.998845 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-socket\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.999197 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgf9\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-kube-api-access-sxgf9\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.999197 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.998910    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-data\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.999197 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.999050    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-certs\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.999349 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.999252    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-credential-socket\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.999349 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.999292    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istiod-ca-cert\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:06.999349 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:06.999296    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-socket\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:07.001038 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.001016    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-envoy\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:07.001268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.001251    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-podinfo\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:07.006336 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.006308    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-token\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:07.006480 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.006459    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgf9\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-kube-api-access-sxgf9\") pod \"e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") " pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:07.142231 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.142149    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:07.271807 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.271777    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"]
Aug 01 01:19:07.273655 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:19:07.273620    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30eedf06_f7aa_4935_8ba5_e54dc7310d12.slice/crio-989f5f0d6933e14de8f39313d1b2691b3f18ababe9bae75da531ed5bac04d258 WatchSource:0}: Error finding container 989f5f0d6933e14de8f39313d1b2691b3f18ababe9bae75da531ed5bac04d258: Status 404 returned error can't find the container with id 989f5f0d6933e14de8f39313d1b2691b3f18ababe9bae75da531ed5bac04d258
Aug 01 01:19:07.276008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.275967    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:19:07.276112 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.276054    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:19:07.276112 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:07.276095    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:19:08.242256 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:08.242219    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" event={"ID":"30eedf06-f7aa-4935-8ba5-e54dc7310d12","Type":"ContainerStarted","Data":"dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496"}
Aug 01 01:19:08.242662 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:08.242274    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" event={"ID":"30eedf06-f7aa-4935-8ba5-e54dc7310d12","Type":"ContainerStarted","Data":"989f5f0d6933e14de8f39313d1b2691b3f18ababe9bae75da531ed5bac04d258"}
Aug 01 01:19:08.263001 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:08.262954    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podStartSLOduration=2.26292413 podStartE2EDuration="2.26292413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:19:08.261372228 +0000 UTC m=+2714.495304508" watchObservedRunningTime="2026-08-01 01:19:08.26292413 +0000 UTC m=+2714.496856396"
Aug 01 01:19:09.142499 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:09.142464    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:09.147251 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:09.147225    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:09.246873 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:09.246845    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:09.247783 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:09.247758    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:11.281891 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.281859    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"]
Aug 01 01:19:11.285579 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.285558    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.288049 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.288033    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1cf076-serving-cert\""
Aug 01 01:19:11.292879 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.292856    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"]
Aug 01 01:19:11.435668 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.435632    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bccb50-1d2d-44f8-8623-21cba46d81cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.435868 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.435697    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bccb50-1d2d-44f8-8623-21cba46d81cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.435868 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.435769    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghglm\" (UniqueName: \"kubernetes.io/projected/50bccb50-1d2d-44f8-8623-21cba46d81cb-kube-api-access-ghglm\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.537139 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.537050    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghglm\" (UniqueName: \"kubernetes.io/projected/50bccb50-1d2d-44f8-8623-21cba46d81cb-kube-api-access-ghglm\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.537139 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.537096    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bccb50-1d2d-44f8-8623-21cba46d81cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.537372 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.537145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bccb50-1d2d-44f8-8623-21cba46d81cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.537852 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.537822    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bccb50-1d2d-44f8-8623-21cba46d81cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.539719 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.539685    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bccb50-1d2d-44f8-8623-21cba46d81cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.545579 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.545559    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghglm\" (UniqueName: \"kubernetes.io/projected/50bccb50-1d2d-44f8-8623-21cba46d81cb-kube-api-access-ghglm\") pod \"maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.597784 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.597760    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:11.733863 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.733834    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"]
Aug 01 01:19:11.736207 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:19:11.736172    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bccb50_1d2d_44f8_8623_21cba46d81cb.slice/crio-c4c426cb903708ce8018a3d80df9c0d299b5c4f886b52fb959b0f562b2ed437e WatchSource:0}: Error finding container c4c426cb903708ce8018a3d80df9c0d299b5c4f886b52fb959b0f562b2ed437e: Status 404 returned error can't find the container with id c4c426cb903708ce8018a3d80df9c0d299b5c4f886b52fb959b0f562b2ed437e
Aug 01 01:19:11.960843 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.960806    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"]
Aug 01 01:19:11.965367 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.965347    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:11.967907 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.967884    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1cf076\""
Aug 01 01:19:11.968084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.968066    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1cf076-dockercfg-z8m9f\""
Aug 01 01:19:11.973696 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:11.973674    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"]
Aug 01 01:19:12.041232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.041201    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnc9\" (UniqueName: \"kubernetes.io/projected/1c15afc9-4d27-42d9-8615-47b6fe717cbb-kube-api-access-jjnc9\") pod \"payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p\" (UID: \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:12.041395 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.041252    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c15afc9-4d27-42d9-8615-47b6fe717cbb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p\" (UID: \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:12.142319 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.142290    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnc9\" (UniqueName: \"kubernetes.io/projected/1c15afc9-4d27-42d9-8615-47b6fe717cbb-kube-api-access-jjnc9\") pod \"payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p\" (UID: \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:12.142462 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.142347    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c15afc9-4d27-42d9-8615-47b6fe717cbb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p\" (UID: \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:12.143084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.143065    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c15afc9-4d27-42d9-8615-47b6fe717cbb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p\" (UID: \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:12.151311 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.151289    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnc9\" (UniqueName: \"kubernetes.io/projected/1c15afc9-4d27-42d9-8615-47b6fe717cbb-kube-api-access-jjnc9\") pod \"payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p\" (UID: \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:12.263003 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.262918    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:12.263153 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.263006    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw" event={"ID":"50bccb50-1d2d-44f8-8623-21cba46d81cb","Type":"ContainerStarted","Data":"56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447"}
Aug 01 01:19:12.263153 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.263039    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw" event={"ID":"50bccb50-1d2d-44f8-8623-21cba46d81cb","Type":"ContainerStarted","Data":"c4c426cb903708ce8018a3d80df9c0d299b5c4f886b52fb959b0f562b2ed437e"}
Aug 01 01:19:12.279122 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.279069    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw" podStartSLOduration=0.94486167 podStartE2EDuration="1.279033157s" totalImagesPullingTime="334.171487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:19:12.277752572 +0000 UTC m=+2718.511684838" watchObservedRunningTime="2026-08-01 01:19:12.279033157 +0000 UTC m=+2718.512965460"
Aug 01 01:19:12.280843 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.280819    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:12.415603 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:12.415576    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"]
Aug 01 01:19:12.417246 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:19:12.417203    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c15afc9_4d27_42d9_8615_47b6fe717cbb.slice/crio-3075620438ab5353f0221f5b5c341964f4d13e71ae7e100dbb4440f0624ce6cf WatchSource:0}: Error finding container 3075620438ab5353f0221f5b5c341964f4d13e71ae7e100dbb4440f0624ce6cf: Status 404 returned error can't find the container with id 3075620438ab5353f0221f5b5c341964f4d13e71ae7e100dbb4440f0624ce6cf
Aug 01 01:19:13.269050 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:13.269010    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p" event={"ID":"1c15afc9-4d27-42d9-8615-47b6fe717cbb","Type":"ContainerStarted","Data":"ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806"}
Aug 01 01:19:13.269273 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:13.269068    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p" event={"ID":"1c15afc9-4d27-42d9-8615-47b6fe717cbb","Type":"ContainerStarted","Data":"3075620438ab5353f0221f5b5c341964f4d13e71ae7e100dbb4440f0624ce6cf"}
Aug 01 01:19:13.285882 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:13.285818    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p" podStartSLOduration=2.2858020310000002 podStartE2EDuration="2.285802031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:19:13.284370701 +0000 UTC m=+2719.518302970" watchObservedRunningTime="2026-08-01 01:19:13.285802031 +0000 UTC m=+2719.519734299"
Aug 01 01:19:14.273210 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:14.273175    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:19.280253 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:19.280216    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:19:24.661468 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:24.661431    2570 patch_prober.go:28] interesting pod/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": read tcp 10.133.0.2:46044->10.133.0.71:15021: read: connection reset by peer" start-of-body=
Aug 01 01:19:24.661872 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:24.661581    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": read tcp 10.133.0.2:46044->10.133.0.71:15021: read: connection reset by peer"
Aug 01 01:19:25.280749 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:25.280718    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:19:25.331985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:25.331953    2570 generic.go:356] "Generic (PLEG): container finished" podID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerID="dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496" exitCode=0
Aug 01 01:19:25.332141 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:25.332034    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" event={"ID":"30eedf06-f7aa-4935-8ba5-e54dc7310d12","Type":"ContainerDied","Data":"dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496"}
Aug 01 01:19:25.332492 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:25.332476    2570 scope.go:117] "RemoveContainer" containerID="dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496"
Aug 01 01:19:25.333569 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:25.333545    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:19:25.333629 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:25.333611    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:19:25.333664 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:25.333652    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:19:26.338487 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:26.338448    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" event={"ID":"30eedf06-f7aa-4935-8ba5-e54dc7310d12","Type":"ContainerStarted","Data":"37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2"}
Aug 01 01:19:27.143079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:27.143047    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:27.143321 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:27.143299    2570 patch_prober.go:28] interesting pod/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Aug 01 01:19:27.143370 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:27.143347    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 01:19:28.142598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:28.142561    2570 patch_prober.go:28] interesting pod/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Aug 01 01:19:28.143012 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:28.142618    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 01:19:29.142991 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:29.142957    2570 patch_prober.go:28] interesting pod/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Aug 01 01:19:29.143366 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:29.143012    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 01:19:30.143465 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:30.143430    2570 patch_prober.go:28] interesting pod/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Aug 01 01:19:30.143858 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:30.143495    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 01:19:31.143246 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:31.143211    2570 patch_prober.go:28] interesting pod/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Aug 01 01:19:31.143427 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:31.143275    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 01:19:32.146653 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:32.146620    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:32.147140 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:32.146908    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:32.147618 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:32.147598    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:19:44.298228 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.298188    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"]
Aug 01 01:19:44.301821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.301798    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.305317 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.305294    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f6a6f5\"/\"openshift-service-ca.crt\""
Aug 01 01:19:44.305317 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.305310    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f6a6f5\"/\"default-dockercfg-8v27p\""
Aug 01 01:19:44.305496 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.305297    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f6a6f5\"/\"test-model-f6a6f5-kserve-self-signed-certs\""
Aug 01 01:19:44.305496 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.305298    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f6a6f5\"/\"kube-root-ca.crt\""
Aug 01 01:19:44.310169 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.310150    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"]
Aug 01 01:19:44.313768 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.313745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-dshm\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.313888 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.313773    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tls-certs\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.314010 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.313984    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tmp-dir\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.314083 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.314025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqc4s\" (UniqueName: \"kubernetes.io/projected/a2930de5-e356-4f0d-af4c-31fa621e0a5b-kube-api-access-fqc4s\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.314143 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.314104    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-home\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.314201 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.314157    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-model-cache\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415114 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415088    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-dshm\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415249 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415118    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tls-certs\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415249 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415201    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tmp-dir\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415249 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415229    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqc4s\" (UniqueName: \"kubernetes.io/projected/a2930de5-e356-4f0d-af4c-31fa621e0a5b-kube-api-access-fqc4s\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415418 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415270    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-home\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415418 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415298    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-model-cache\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415593 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415570    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tmp-dir\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415665 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415618    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-home\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.415665 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.415644    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-model-cache\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.417335 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.417315    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-dshm\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.417618 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.417597    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tls-certs\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.422498 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.422469    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqc4s\" (UniqueName: \"kubernetes.io/projected/a2930de5-e356-4f0d-af4c-31fa621e0a5b-kube-api-access-fqc4s\") pod \"test-model-f6a6f5-kserve-789db65565-l4wgk\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") " pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.611840 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.611813    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:44.745299 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:44.745266    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"]
Aug 01 01:19:44.748614 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:19:44.748586    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2930de5_e356_4f0d_af4c_31fa621e0a5b.slice/crio-33ff83ea43d9a624174261c1601713bddee5a44e25e99429562e801a480e4cc1 WatchSource:0}: Error finding container 33ff83ea43d9a624174261c1601713bddee5a44e25e99429562e801a480e4cc1: Status 404 returned error can't find the container with id 33ff83ea43d9a624174261c1601713bddee5a44e25e99429562e801a480e4cc1
Aug 01 01:19:45.436028 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:45.435995    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk" event={"ID":"a2930de5-e356-4f0d-af4c-31fa621e0a5b","Type":"ContainerStarted","Data":"b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b"}
Aug 01 01:19:45.436394 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:45.436034    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk" event={"ID":"a2930de5-e356-4f0d-af4c-31fa621e0a5b","Type":"ContainerStarted","Data":"33ff83ea43d9a624174261c1601713bddee5a44e25e99429562e801a480e4cc1"}
Aug 01 01:19:45.454909 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:45.454864    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk" podStartSLOduration=1.45484876 podStartE2EDuration="1.45484876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:19:45.453274352 +0000 UTC m=+2751.687206612" watchObservedRunningTime="2026-08-01 01:19:45.45484876 +0000 UTC m=+2751.688781026"
Aug 01 01:19:54.612864 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:54.612835    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:54.613327 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:54.612881    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:54.625394 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:54.625369    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:19:55.495251 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:19:55.495221    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:20:18.497292 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.497211    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"]
Aug 01 01:20:18.500903 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.500882    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.504568 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.504544    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1cf076\"/\"test-model-1cf076-kserve-self-signed-certs\""
Aug 01 01:20:18.504703 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.504571    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1cf076\"/\"openshift-service-ca.crt\""
Aug 01 01:20:18.504703 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.504571    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1cf076\"/\"kube-root-ca.crt\""
Aug 01 01:20:18.504703 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.504584    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1cf076\"/\"default-dockercfg-8rn47\""
Aug 01 01:20:18.511417 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.511368    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"]
Aug 01 01:20:18.606584 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.606556    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-model-cache\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.606714 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.606602    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-home\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.606714 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.606626    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjg92\" (UniqueName: \"kubernetes.io/projected/40fa87cc-f5b6-4599-9774-91c3bda05fe1-kube-api-access-tjg92\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.606714 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.606686    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-dshm\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.606826 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.606714    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tmp-dir\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.606826 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.606737    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tls-certs\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707279 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707244    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-model-cache\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707318    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-home\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjg92\" (UniqueName: \"kubernetes.io/projected/40fa87cc-f5b6-4599-9774-91c3bda05fe1-kube-api-access-tjg92\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707369    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-dshm\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707389    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tmp-dir\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707409    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tls-certs\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707718 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707652    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-model-cache\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707776 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707714    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-home\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.707831 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.707775    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tmp-dir\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.709702 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.709681    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-dshm\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.709879 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.709862    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tls-certs\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.715708 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.715681    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjg92\" (UniqueName: \"kubernetes.io/projected/40fa87cc-f5b6-4599-9774-91c3bda05fe1-kube-api-access-tjg92\") pod \"test-model-1cf076-kserve-6bb6fc67c5-b9f8q\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") " pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.813568 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.813492    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:18.942605 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:18.942577    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"]
Aug 01 01:20:18.944942 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:20:18.944904    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40fa87cc_f5b6_4599_9774_91c3bda05fe1.slice/crio-687523ef9d9ec9ca2f5e62d69eb670a40665538f4fb48ec4f78f5f24d936f913 WatchSource:0}: Error finding container 687523ef9d9ec9ca2f5e62d69eb670a40665538f4fb48ec4f78f5f24d936f913: Status 404 returned error can't find the container with id 687523ef9d9ec9ca2f5e62d69eb670a40665538f4fb48ec4f78f5f24d936f913
Aug 01 01:20:19.601486 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:19.601450    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q" event={"ID":"40fa87cc-f5b6-4599-9774-91c3bda05fe1","Type":"ContainerStarted","Data":"87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b"}
Aug 01 01:20:19.601887 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:19.601500    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q" event={"ID":"40fa87cc-f5b6-4599-9774-91c3bda05fe1","Type":"ContainerStarted","Data":"687523ef9d9ec9ca2f5e62d69eb670a40665538f4fb48ec4f78f5f24d936f913"}
Aug 01 01:20:19.618260 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:19.618210    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q" podStartSLOduration=1.61819501 podStartE2EDuration="1.61819501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:20:19.61678977 +0000 UTC m=+2785.850722037" watchObservedRunningTime="2026-08-01 01:20:19.61819501 +0000 UTC m=+2785.852127276"
Aug 01 01:20:28.813737 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:28.813699    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:28.814229 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:28.813751    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:28.826193 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:28.826165    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:20:29.660320 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:20:29.660290    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:21:10.498580 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:10.498537    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"]
Aug 01 01:21:11.398339 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:11.398305    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"]
Aug 01 01:21:21.879516 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:21.879478    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"]
Aug 01 01:21:21.880020 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:21.879751    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc" podUID="45251bd8-26de-4a44-953a-f5e9b4f0e629" containerName="maas-api" containerID="cri-o://ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193" gracePeriod=30
Aug 01 01:21:21.944887 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:21.944857    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"]
Aug 01 01:21:21.945120 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:21.945091    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x" podUID="6be06045-5456-43de-be0b-a37afb466886" containerName="payload-pre-processing" containerID="cri-o://55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5" gracePeriod=30
Aug 01 01:21:22.223201 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.223177    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:21:22.345441 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.345418    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:21:22.346528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.346510    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be06045-5456-43de-be0b-a37afb466886-plugins-config-volume\") pod \"6be06045-5456-43de-be0b-a37afb466886\" (UID: \"6be06045-5456-43de-be0b-a37afb466886\") "
Aug 01 01:21:22.346612 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.346600    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb8jt\" (UniqueName: \"kubernetes.io/projected/6be06045-5456-43de-be0b-a37afb466886-kube-api-access-rb8jt\") pod \"6be06045-5456-43de-be0b-a37afb466886\" (UID: \"6be06045-5456-43de-be0b-a37afb466886\") "
Aug 01 01:21:22.346908 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.346878    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be06045-5456-43de-be0b-a37afb466886-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6be06045-5456-43de-be0b-a37afb466886" (UID: "6be06045-5456-43de-be0b-a37afb466886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:21:22.348549 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.348493    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be06045-5456-43de-be0b-a37afb466886-kube-api-access-rb8jt" (OuterVolumeSpecName: "kube-api-access-rb8jt") pod "6be06045-5456-43de-be0b-a37afb466886" (UID: "6be06045-5456-43de-be0b-a37afb466886"). InnerVolumeSpecName "kube-api-access-rb8jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:22.447332 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.447260    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls\") pod \"45251bd8-26de-4a44-953a-f5e9b4f0e629\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") "
Aug 01 01:21:22.447332 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.447309    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gntz\" (UniqueName: \"kubernetes.io/projected/45251bd8-26de-4a44-953a-f5e9b4f0e629-kube-api-access-7gntz\") pod \"45251bd8-26de-4a44-953a-f5e9b4f0e629\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") "
Aug 01 01:21:22.447552 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.447352    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45251bd8-26de-4a44-953a-f5e9b4f0e629-openshift-service-ca\") pod \"45251bd8-26de-4a44-953a-f5e9b4f0e629\" (UID: \"45251bd8-26de-4a44-953a-f5e9b4f0e629\") "
Aug 01 01:21:22.447552 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.447548    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be06045-5456-43de-be0b-a37afb466886-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:22.447660 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.447559    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8jt\" (UniqueName: \"kubernetes.io/projected/6be06045-5456-43de-be0b-a37afb466886-kube-api-access-rb8jt\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:22.447764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.447740    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45251bd8-26de-4a44-953a-f5e9b4f0e629-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45251bd8-26de-4a44-953a-f5e9b4f0e629" (UID: "45251bd8-26de-4a44-953a-f5e9b4f0e629"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:21:22.449252 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.449226    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45251bd8-26de-4a44-953a-f5e9b4f0e629-kube-api-access-7gntz" (OuterVolumeSpecName: "kube-api-access-7gntz") pod "45251bd8-26de-4a44-953a-f5e9b4f0e629" (UID: "45251bd8-26de-4a44-953a-f5e9b4f0e629"). InnerVolumeSpecName "kube-api-access-7gntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:22.449343 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.449253    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45251bd8-26de-4a44-953a-f5e9b4f0e629" (UID: "45251bd8-26de-4a44-953a-f5e9b4f0e629"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:21:22.548424 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.548399    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45251bd8-26de-4a44-953a-f5e9b4f0e629-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:22.548424 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.548422    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gntz\" (UniqueName: \"kubernetes.io/projected/45251bd8-26de-4a44-953a-f5e9b4f0e629-kube-api-access-7gntz\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:22.548580 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.548433    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45251bd8-26de-4a44-953a-f5e9b4f0e629-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:22.912807 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.912769    2570 generic.go:356] "Generic (PLEG): container finished" podID="6be06045-5456-43de-be0b-a37afb466886" containerID="55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5" exitCode=0
Aug 01 01:21:22.913268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.912836    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"
Aug 01 01:21:22.913268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.912854    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x" event={"ID":"6be06045-5456-43de-be0b-a37afb466886","Type":"ContainerDied","Data":"55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5"}
Aug 01 01:21:22.913268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.912969    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x" event={"ID":"6be06045-5456-43de-be0b-a37afb466886","Type":"ContainerDied","Data":"bd18a72eabbe8d1869fdc08d44e8a049aaf9737182ea747df8dc00baa9aedeca"}
Aug 01 01:21:22.913268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.913003    2570 scope.go:117] "RemoveContainer" containerID="55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5"
Aug 01 01:21:22.914336 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.914211    2570 generic.go:356] "Generic (PLEG): container finished" podID="45251bd8-26de-4a44-953a-f5e9b4f0e629" containerID="ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193" exitCode=0
Aug 01 01:21:22.914336 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.914251    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc" event={"ID":"45251bd8-26de-4a44-953a-f5e9b4f0e629","Type":"ContainerDied","Data":"ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193"}
Aug 01 01:21:22.914336 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.914265    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"
Aug 01 01:21:22.914336 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.914293    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc" event={"ID":"45251bd8-26de-4a44-953a-f5e9b4f0e629","Type":"ContainerDied","Data":"8cfa4a6efaa9126f8f06fc81b081c150756eb54b5b1fdfb0f0080d981c8835ee"}
Aug 01 01:21:22.925014 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.924980    2570 scope.go:117] "RemoveContainer" containerID="55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5"
Aug 01 01:21:22.925297 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:22.925269    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5\": container with ID starting with 55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5 not found: ID does not exist" containerID="55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5"
Aug 01 01:21:22.925400 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.925305    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5"} err="failed to get container status \"55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5\": rpc error: code = NotFound desc = could not find container \"55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5\": container with ID starting with 55d25398f3a5504453d275e28ac3e484a22ad81f45e7c6c909f13705607dd6a5 not found: ID does not exist"
Aug 01 01:21:22.925400 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.925329    2570 scope.go:117] "RemoveContainer" containerID="ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193"
Aug 01 01:21:22.933567 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.933550    2570 scope.go:117] "RemoveContainer" containerID="ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193"
Aug 01 01:21:22.933810 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:22.933793    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193\": container with ID starting with ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193 not found: ID does not exist" containerID="ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193"
Aug 01 01:21:22.933856 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.933816    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193"} err="failed to get container status \"ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193\": rpc error: code = NotFound desc = could not find container \"ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193\": container with ID starting with ab2cc5446e3487673d169b4a9dbe2358649c664163fa9baf75d60ea99dc95193 not found: ID does not exist"
Aug 01 01:21:22.940366 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.940344    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"]
Aug 01 01:21:22.943845 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.943825    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f6a6f5-594d4978f6-54bxc"]
Aug 01 01:21:22.954208 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.954187    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"]
Aug 01 01:21:22.955977 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:22.955955    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f6a6f5-7ff98c89d8-pv77x"]
Aug 01 01:21:24.263570 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:24.263536    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45251bd8-26de-4a44-953a-f5e9b4f0e629" path="/var/lib/kubelet/pods/45251bd8-26de-4a44-953a-f5e9b4f0e629/volumes"
Aug 01 01:21:24.264089 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:24.263959    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be06045-5456-43de-be0b-a37afb466886" path="/var/lib/kubelet/pods/6be06045-5456-43de-be0b-a37afb466886/volumes"
Aug 01 01:21:25.517890 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.517847    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk" podUID="a2930de5-e356-4f0d-af4c-31fa621e0a5b" containerName="main" containerID="cri-o://b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b" gracePeriod=45
Aug 01 01:21:25.769897 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.769844    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:21:25.877439 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877411    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tmp-dir\") pod \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") "
Aug 01 01:21:25.877571 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877471    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tls-certs\") pod \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") "
Aug 01 01:21:25.877571 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877513    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-model-cache\") pod \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") "
Aug 01 01:21:25.877571 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877541    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-dshm\") pod \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") "
Aug 01 01:21:25.877678 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877588    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqc4s\" (UniqueName: \"kubernetes.io/projected/a2930de5-e356-4f0d-af4c-31fa621e0a5b-kube-api-access-fqc4s\") pod \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") "
Aug 01 01:21:25.877678 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877607    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2930de5-e356-4f0d-af4c-31fa621e0a5b" (UID: "a2930de5-e356-4f0d-af4c-31fa621e0a5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:25.877678 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877621    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-home\") pod \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\" (UID: \"a2930de5-e356-4f0d-af4c-31fa621e0a5b\") "
Aug 01 01:21:25.877817 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877781    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2930de5-e356-4f0d-af4c-31fa621e0a5b" (UID: "a2930de5-e356-4f0d-af4c-31fa621e0a5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:25.877984 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877923    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-home" (OuterVolumeSpecName: "home") pod "a2930de5-e356-4f0d-af4c-31fa621e0a5b" (UID: "a2930de5-e356-4f0d-af4c-31fa621e0a5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:25.878109 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.877965    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:25.878109 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.878013    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:25.879773 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.879753    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2930de5-e356-4f0d-af4c-31fa621e0a5b-kube-api-access-fqc4s" (OuterVolumeSpecName: "kube-api-access-fqc4s") pod "a2930de5-e356-4f0d-af4c-31fa621e0a5b" (UID: "a2930de5-e356-4f0d-af4c-31fa621e0a5b"). InnerVolumeSpecName "kube-api-access-fqc4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:25.880042 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.880018    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2930de5-e356-4f0d-af4c-31fa621e0a5b" (UID: "a2930de5-e356-4f0d-af4c-31fa621e0a5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:21:25.880132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.880038    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-dshm" (OuterVolumeSpecName: "dshm") pod "a2930de5-e356-4f0d-af4c-31fa621e0a5b" (UID: "a2930de5-e356-4f0d-af4c-31fa621e0a5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:25.930911 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.930882    2570 generic.go:356] "Generic (PLEG): container finished" podID="a2930de5-e356-4f0d-af4c-31fa621e0a5b" containerID="b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b" exitCode=0
Aug 01 01:21:25.931008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.930920    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"
Aug 01 01:21:25.931008 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.930968    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk" event={"ID":"a2930de5-e356-4f0d-af4c-31fa621e0a5b","Type":"ContainerDied","Data":"b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b"}
Aug 01 01:21:25.931083 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.931011    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk" event={"ID":"a2930de5-e356-4f0d-af4c-31fa621e0a5b","Type":"ContainerDied","Data":"33ff83ea43d9a624174261c1601713bddee5a44e25e99429562e801a480e4cc1"}
Aug 01 01:21:25.931083 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.931031    2570 scope.go:117] "RemoveContainer" containerID="b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b"
Aug 01 01:21:25.940308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.940289    2570 scope.go:117] "RemoveContainer" containerID="b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b"
Aug 01 01:21:25.940561 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:25.940538    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b\": container with ID starting with b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b not found: ID does not exist" containerID="b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b"
Aug 01 01:21:25.940623 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.940569    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b"} err="failed to get container status \"b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b\": rpc error: code = NotFound desc = could not find container \"b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b\": container with ID starting with b5bbfaaddb53ea3bc591c96b739e56a2189fda0c6f14e16791e68931d664450b not found: ID does not exist"
Aug 01 01:21:25.951862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.951840    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"]
Aug 01 01:21:25.955086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.955063    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f6a6f5/test-model-f6a6f5-kserve-789db65565-l4wgk"]
Aug 01 01:21:25.979102 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.979082    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2930de5-e356-4f0d-af4c-31fa621e0a5b-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:25.979102 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.979099    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:25.979216 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.979109    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqc4s\" (UniqueName: \"kubernetes.io/projected/a2930de5-e356-4f0d-af4c-31fa621e0a5b-kube-api-access-fqc4s\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:25.979216 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:25.979118    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2930de5-e356-4f0d-af4c-31fa621e0a5b-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:26.261743 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.261713    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2930de5-e356-4f0d-af4c-31fa621e0a5b" path="/var/lib/kubelet/pods/a2930de5-e356-4f0d-af4c-31fa621e0a5b/volumes"
Aug 01 01:21:26.418304 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.418275    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q" podUID="40fa87cc-f5b6-4599-9774-91c3bda05fe1" containerName="main" containerID="cri-o://87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b" gracePeriod=45
Aug 01 01:21:26.662043 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.662019    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:21:26.785985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.785893    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tmp-dir\") pod \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") "
Aug 01 01:21:26.785985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.785969    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjg92\" (UniqueName: \"kubernetes.io/projected/40fa87cc-f5b6-4599-9774-91c3bda05fe1-kube-api-access-tjg92\") pod \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") "
Aug 01 01:21:26.786200 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.786001    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-dshm\") pod \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") "
Aug 01 01:21:26.786200 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.786017    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tls-certs\") pod \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") "
Aug 01 01:21:26.786200 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.786066    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-model-cache\") pod \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") "
Aug 01 01:21:26.786200 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.786085    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-home\") pod \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\" (UID: \"40fa87cc-f5b6-4599-9774-91c3bda05fe1\") "
Aug 01 01:21:26.786401 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.786188    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40fa87cc-f5b6-4599-9774-91c3bda05fe1" (UID: "40fa87cc-f5b6-4599-9774-91c3bda05fe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:26.786401 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.786333    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:26.786401 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.786370    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-model-cache" (OuterVolumeSpecName: "model-cache") pod "40fa87cc-f5b6-4599-9774-91c3bda05fe1" (UID: "40fa87cc-f5b6-4599-9774-91c3bda05fe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:26.786401 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.786379    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-home" (OuterVolumeSpecName: "home") pod "40fa87cc-f5b6-4599-9774-91c3bda05fe1" (UID: "40fa87cc-f5b6-4599-9774-91c3bda05fe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:26.788231 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.788202    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-dshm" (OuterVolumeSpecName: "dshm") pod "40fa87cc-f5b6-4599-9774-91c3bda05fe1" (UID: "40fa87cc-f5b6-4599-9774-91c3bda05fe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:26.788231 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.788206    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40fa87cc-f5b6-4599-9774-91c3bda05fe1" (UID: "40fa87cc-f5b6-4599-9774-91c3bda05fe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:21:26.788381 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.788282    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fa87cc-f5b6-4599-9774-91c3bda05fe1-kube-api-access-tjg92" (OuterVolumeSpecName: "kube-api-access-tjg92") pod "40fa87cc-f5b6-4599-9774-91c3bda05fe1" (UID: "40fa87cc-f5b6-4599-9774-91c3bda05fe1"). InnerVolumeSpecName "kube-api-access-tjg92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:26.887346 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.887320    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjg92\" (UniqueName: \"kubernetes.io/projected/40fa87cc-f5b6-4599-9774-91c3bda05fe1-kube-api-access-tjg92\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:26.887346 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.887342    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:26.887498 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.887355    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fa87cc-f5b6-4599-9774-91c3bda05fe1-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:26.887498 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.887363    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:26.887498 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.887371    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fa87cc-f5b6-4599-9774-91c3bda05fe1-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:26.943191 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.943163    2570 generic.go:356] "Generic (PLEG): container finished" podID="40fa87cc-f5b6-4599-9774-91c3bda05fe1" containerID="87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b" exitCode=0
Aug 01 01:21:26.943308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.943214    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"
Aug 01 01:21:26.943308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.943254    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q" event={"ID":"40fa87cc-f5b6-4599-9774-91c3bda05fe1","Type":"ContainerDied","Data":"87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b"}
Aug 01 01:21:26.943308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.943304    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q" event={"ID":"40fa87cc-f5b6-4599-9774-91c3bda05fe1","Type":"ContainerDied","Data":"687523ef9d9ec9ca2f5e62d69eb670a40665538f4fb48ec4f78f5f24d936f913"}
Aug 01 01:21:26.943457 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.943327    2570 scope.go:117] "RemoveContainer" containerID="87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b"
Aug 01 01:21:26.951998 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.951976    2570 scope.go:117] "RemoveContainer" containerID="87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b"
Aug 01 01:21:26.952247 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:26.952229    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b\": container with ID starting with 87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b not found: ID does not exist" containerID="87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b"
Aug 01 01:21:26.952292 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.952255    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b"} err="failed to get container status \"87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b\": rpc error: code = NotFound desc = could not find container \"87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b\": container with ID starting with 87c7214c5a42aea4525d5d208d9355229fce7160c3686b27abd701e311db441b not found: ID does not exist"
Aug 01 01:21:26.963005 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.962984    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"]
Aug 01 01:21:26.968415 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:26.968395    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1cf076/test-model-1cf076-kserve-6bb6fc67c5-b9f8q"]
Aug 01 01:21:28.262558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:28.262525    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40fa87cc-f5b6-4599-9774-91c3bda05fe1" path="/var/lib/kubelet/pods/40fa87cc-f5b6-4599-9774-91c3bda05fe1/volumes"
Aug 01 01:21:28.438264 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:28.438229    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"]
Aug 01 01:21:28.438513 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:28.438483    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy" containerID="cri-o://98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668" gracePeriod=30
Aug 01 01:21:31.409677 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:31.409644    2570 patch_prober.go:28] interesting pod/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Aug 01 01:21:31.410066 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:31.409700    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 01:21:33.688825 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.688802    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:21:33.848642 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848612    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istiod-ca-cert\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.848642 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848645    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-token\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.848850 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848671    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-envoy\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.848850 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848722    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-data\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.848850 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848740    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-certs\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.848850 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848769    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csgbl\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-kube-api-access-csgbl\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.848850 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848826    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-socket\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.849140 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848856    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-credential-socket\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.849140 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.848880    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-podinfo\") pod \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\" (UID: \"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0\") "
Aug 01 01:21:33.849140 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.849084    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-data" (OuterVolumeSpecName: "istio-data") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:33.849140 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.849098    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:21:33.849140 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.849103    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:33.849381 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.849235    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:33.849381 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.849301    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.849381 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.849323    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.849381 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.849338    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.849602 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.849550    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:33.850997 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.850965    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-token" (OuterVolumeSpecName: "istio-token") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:33.851118 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.850999    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:33.851118 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.851051    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-kube-api-access-csgbl" (OuterVolumeSpecName: "kube-api-access-csgbl") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "kube-api-access-csgbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:33.851118 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.851086    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" (UID: "75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:21:33.950558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.950532    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.950558 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.950554    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.950734 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.950566    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csgbl\" (UniqueName: \"kubernetes.io/projected/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-kube-api-access-csgbl\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.950734 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.950582    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.950734 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.950594    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.950734 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.950607    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:33.979179 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.979154    2570 generic.go:356] "Generic (PLEG): container finished" podID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerID="98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668" exitCode=0
Aug 01 01:21:33.979308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.979189    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" event={"ID":"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0","Type":"ContainerDied","Data":"98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668"}
Aug 01 01:21:33.979308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.979209    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"
Aug 01 01:21:33.979308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.979259    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f" event={"ID":"75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0","Type":"ContainerDied","Data":"ef0a326396975ec7e83d8625a2f225a2c3ae71614b076fd78b07b55b144b9b16"}
Aug 01 01:21:33.979308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.979279    2570 scope.go:117] "RemoveContainer" containerID="98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668"
Aug 01 01:21:33.988152 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.988135    2570 scope.go:117] "RemoveContainer" containerID="1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e"
Aug 01 01:21:33.995756 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.995741    2570 scope.go:117] "RemoveContainer" containerID="98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668"
Aug 01 01:21:33.995998 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:33.995980    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668\": container with ID starting with 98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668 not found: ID does not exist" containerID="98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668"
Aug 01 01:21:33.996045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.996008    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668"} err="failed to get container status \"98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668\": rpc error: code = NotFound desc = could not find container \"98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668\": container with ID starting with 98cfdfdfa12d2ffc85a04e4eb32a937dca99d13f75e109b66754a84de2262668 not found: ID does not exist"
Aug 01 01:21:33.996045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.996024    2570 scope.go:117] "RemoveContainer" containerID="1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e"
Aug 01 01:21:33.996263 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:33.996245    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e\": container with ID starting with 1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e not found: ID does not exist" containerID="1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e"
Aug 01 01:21:33.996300 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:33.996271    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e"} err="failed to get container status \"1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e\": rpc error: code = NotFound desc = could not find container \"1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e\": container with ID starting with 1e3b9ba362fce96e7949070dc9165cbd1c226654d40a2c878d9b58b96a0dad5e not found: ID does not exist"
Aug 01 01:21:34.000227 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:34.000208    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"]
Aug 01 01:21:34.003915 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:34.003891    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f6a6f5-openshift-default-679cbbff74-n8p9f"]
Aug 01 01:21:34.262756 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:34.262681    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" path="/var/lib/kubelet/pods/75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0/volumes"
Aug 01 01:21:39.162382 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.162350    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"]
Aug 01 01:21:39.162832 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.162614    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw" podUID="50bccb50-1d2d-44f8-8623-21cba46d81cb" containerName="maas-api" containerID="cri-o://56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447" gracePeriod=30
Aug 01 01:21:39.214834 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.214803    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"]
Aug 01 01:21:39.215094 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.215068    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p" podUID="1c15afc9-4d27-42d9-8615-47b6fe717cbb" containerName="payload-processing" containerID="cri-o://ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806" gracePeriod=30
Aug 01 01:21:39.274158 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.274122    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw" podUID="50bccb50-1d2d-44f8-8623-21cba46d81cb" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.72:8443/health\": dial tcp 10.133.0.72:8443: connect: connection refused"
Aug 01 01:21:39.508495 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.508244    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:21:39.595590 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.595556    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjnc9\" (UniqueName: \"kubernetes.io/projected/1c15afc9-4d27-42d9-8615-47b6fe717cbb-kube-api-access-jjnc9\") pod \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\" (UID: \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\") "
Aug 01 01:21:39.595758 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.595686    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c15afc9-4d27-42d9-8615-47b6fe717cbb-plugins-config-volume\") pod \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\" (UID: \"1c15afc9-4d27-42d9-8615-47b6fe717cbb\") "
Aug 01 01:21:39.596103 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.596074    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c15afc9-4d27-42d9-8615-47b6fe717cbb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c15afc9-4d27-42d9-8615-47b6fe717cbb" (UID: "1c15afc9-4d27-42d9-8615-47b6fe717cbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:21:39.597728 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.597698    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c15afc9-4d27-42d9-8615-47b6fe717cbb-kube-api-access-jjnc9" (OuterVolumeSpecName: "kube-api-access-jjnc9") pod "1c15afc9-4d27-42d9-8615-47b6fe717cbb" (UID: "1c15afc9-4d27-42d9-8615-47b6fe717cbb"). InnerVolumeSpecName "kube-api-access-jjnc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:39.618900 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.618879    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:21:39.696396 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.696320    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bccb50-1d2d-44f8-8623-21cba46d81cb-openshift-service-ca\") pod \"50bccb50-1d2d-44f8-8623-21cba46d81cb\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") "
Aug 01 01:21:39.696396 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.696370    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghglm\" (UniqueName: \"kubernetes.io/projected/50bccb50-1d2d-44f8-8623-21cba46d81cb-kube-api-access-ghglm\") pod \"50bccb50-1d2d-44f8-8623-21cba46d81cb\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") "
Aug 01 01:21:39.696577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.696401    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bccb50-1d2d-44f8-8623-21cba46d81cb-maas-api-tls\") pod \"50bccb50-1d2d-44f8-8623-21cba46d81cb\" (UID: \"50bccb50-1d2d-44f8-8623-21cba46d81cb\") "
Aug 01 01:21:39.696577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.696510    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnc9\" (UniqueName: \"kubernetes.io/projected/1c15afc9-4d27-42d9-8615-47b6fe717cbb-kube-api-access-jjnc9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:39.696577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.696521    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c15afc9-4d27-42d9-8615-47b6fe717cbb-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:39.696760 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.696728    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50bccb50-1d2d-44f8-8623-21cba46d81cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50bccb50-1d2d-44f8-8623-21cba46d81cb" (UID: "50bccb50-1d2d-44f8-8623-21cba46d81cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:21:39.698440 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.698416    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50bccb50-1d2d-44f8-8623-21cba46d81cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50bccb50-1d2d-44f8-8623-21cba46d81cb" (UID: "50bccb50-1d2d-44f8-8623-21cba46d81cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:21:39.698543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.698453    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bccb50-1d2d-44f8-8623-21cba46d81cb-kube-api-access-ghglm" (OuterVolumeSpecName: "kube-api-access-ghglm") pod "50bccb50-1d2d-44f8-8623-21cba46d81cb" (UID: "50bccb50-1d2d-44f8-8623-21cba46d81cb"). InnerVolumeSpecName "kube-api-access-ghglm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:39.797225 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.797198    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bccb50-1d2d-44f8-8623-21cba46d81cb-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:39.797225 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.797220    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghglm\" (UniqueName: \"kubernetes.io/projected/50bccb50-1d2d-44f8-8623-21cba46d81cb-kube-api-access-ghglm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:39.797225 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:39.797230    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bccb50-1d2d-44f8-8623-21cba46d81cb-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:40.009251 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.009166    2570 generic.go:356] "Generic (PLEG): container finished" podID="50bccb50-1d2d-44f8-8623-21cba46d81cb" containerID="56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447" exitCode=0
Aug 01 01:21:40.009251 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.009215    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"
Aug 01 01:21:40.009416 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.009249    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw" event={"ID":"50bccb50-1d2d-44f8-8623-21cba46d81cb","Type":"ContainerDied","Data":"56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447"}
Aug 01 01:21:40.009416 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.009301    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw" event={"ID":"50bccb50-1d2d-44f8-8623-21cba46d81cb","Type":"ContainerDied","Data":"c4c426cb903708ce8018a3d80df9c0d299b5c4f886b52fb959b0f562b2ed437e"}
Aug 01 01:21:40.009416 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.009325    2570 scope.go:117] "RemoveContainer" containerID="56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447"
Aug 01 01:21:40.010570 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.010545    2570 generic.go:356] "Generic (PLEG): container finished" podID="1c15afc9-4d27-42d9-8615-47b6fe717cbb" containerID="ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806" exitCode=0
Aug 01 01:21:40.010695 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.010584    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"
Aug 01 01:21:40.010695 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.010626    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p" event={"ID":"1c15afc9-4d27-42d9-8615-47b6fe717cbb","Type":"ContainerDied","Data":"ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806"}
Aug 01 01:21:40.010695 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.010685    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p" event={"ID":"1c15afc9-4d27-42d9-8615-47b6fe717cbb","Type":"ContainerDied","Data":"3075620438ab5353f0221f5b5c341964f4d13e71ae7e100dbb4440f0624ce6cf"}
Aug 01 01:21:40.018678 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.018661    2570 scope.go:117] "RemoveContainer" containerID="56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447"
Aug 01 01:21:40.018979 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:40.018955    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447\": container with ID starting with 56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447 not found: ID does not exist" containerID="56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447"
Aug 01 01:21:40.019096 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.018983    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447"} err="failed to get container status \"56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447\": rpc error: code = NotFound desc = could not find container \"56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447\": container with ID starting with 56989a5cf7ff9b50e53b821b935d307cbfe53ac724270edc1988e1a82ce82447 not found: ID does not exist"
Aug 01 01:21:40.019096 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.019000    2570 scope.go:117] "RemoveContainer" containerID="ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806"
Aug 01 01:21:40.027012 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.026995    2570 scope.go:117] "RemoveContainer" containerID="ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806"
Aug 01 01:21:40.027299 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:40.027281    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806\": container with ID starting with ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806 not found: ID does not exist" containerID="ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806"
Aug 01 01:21:40.027431 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.027304    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806"} err="failed to get container status \"ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806\": rpc error: code = NotFound desc = could not find container \"ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806\": container with ID starting with ac0cd44bfe101e455b0f54b8bca2dd650874b118813d66f3741432045ed5f806 not found: ID does not exist"
Aug 01 01:21:40.031448 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.031426    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"]
Aug 01 01:21:40.033039 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.033021    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1cf076-7c59c75494-q85fw"]
Aug 01 01:21:40.042474 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.042445    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"]
Aug 01 01:21:40.046284 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.046262    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1cf076-79db7ddfbb-phc4p"]
Aug 01 01:21:40.261769 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.261702    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c15afc9-4d27-42d9-8615-47b6fe717cbb" path="/var/lib/kubelet/pods/1c15afc9-4d27-42d9-8615-47b6fe717cbb/volumes"
Aug 01 01:21:40.262104 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:40.262091    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50bccb50-1d2d-44f8-8623-21cba46d81cb" path="/var/lib/kubelet/pods/50bccb50-1d2d-44f8-8623-21cba46d81cb/volumes"
Aug 01 01:21:45.603881 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:45.603847    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"]
Aug 01 01:21:45.604520 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:45.604063    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy" containerID="cri-o://37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2" gracePeriod=30
Aug 01 01:21:46.500724 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.500640    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"]
Aug 01 01:21:46.501118 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501099    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fa87cc-f5b6-4599-9774-91c3bda05fe1" containerName="main"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501120    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fa87cc-f5b6-4599-9774-91c3bda05fe1" containerName="main"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501148    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45251bd8-26de-4a44-953a-f5e9b4f0e629" containerName="maas-api"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501156    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="45251bd8-26de-4a44-953a-f5e9b4f0e629" containerName="maas-api"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501179    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be06045-5456-43de-be0b-a37afb466886" containerName="payload-pre-processing"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501187    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be06045-5456-43de-be0b-a37afb466886" containerName="payload-pre-processing"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501195    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501204    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501214    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50bccb50-1d2d-44f8-8623-21cba46d81cb" containerName="maas-api"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501222    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="50bccb50-1d2d-44f8-8623-21cba46d81cb" containerName="maas-api"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501230    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2930de5-e356-4f0d-af4c-31fa621e0a5b" containerName="main"
Aug 01 01:21:46.501234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501238    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2930de5-e356-4f0d-af4c-31fa621e0a5b" containerName="main"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501249    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c15afc9-4d27-42d9-8615-47b6fe717cbb" containerName="payload-processing"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501256    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c15afc9-4d27-42d9-8615-47b6fe717cbb" containerName="payload-processing"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501274    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501282    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501366    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c15afc9-4d27-42d9-8615-47b6fe717cbb" containerName="payload-processing"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501381    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="45251bd8-26de-4a44-953a-f5e9b4f0e629" containerName="maas-api"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501389    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2930de5-e356-4f0d-af4c-31fa621e0a5b" containerName="main"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501398    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fa87cc-f5b6-4599-9774-91c3bda05fe1" containerName="main"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501408    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501419    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be06045-5456-43de-be0b-a37afb466886" containerName="payload-pre-processing"
Aug 01 01:21:46.501752 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.501428    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="50bccb50-1d2d-44f8-8623-21cba46d81cb" containerName="maas-api"
Aug 01 01:21:46.506052 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.506032    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.508824 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.508804    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-041b9f-openshift-default-dockercfg-s8bn2\""
Aug 01 01:21:46.514547 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.514523    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"]
Aug 01 01:21:46.552543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552512    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-data\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.552672 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552547    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-podinfo\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.552672 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552592    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istiod-ca-cert\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.552672 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552639    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-credential-socket\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.552672 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552663    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-certs\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.552896 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-token\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.552896 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552724    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-envoy\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.552896 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927fn\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-kube-api-access-927fn\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.552896 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.552821    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-socket\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653385 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653348    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-data\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653389    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-podinfo\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653429    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istiod-ca-cert\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-credential-socket\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653507    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-certs\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653538    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-token\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-envoy\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653608    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927fn\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-kube-api-access-927fn\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653659    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-socket\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.653821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653799    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-data\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.654324 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653868    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-certs\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.654324 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.653975    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-credential-socket\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.654324 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.654198    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istiod-ca-cert\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.654324 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.654305    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-socket\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.655838 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.655815    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-envoy\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.655971 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.655951    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-podinfo\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.661811 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.661793    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-token\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.662076 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.662054    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-927fn\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-kube-api-access-927fn\") pod \"e2e-api-a-041b9f-openshift-default-749845db84-fgrbn\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") " pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.818093 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.818065    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:46.972896 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.967983    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"]
Aug 01 01:21:46.974609 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.974574    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:21:46.974698 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.974657    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:21:46.974736 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:46.974693    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:21:47.049216 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:47.049183    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" event={"ID":"7a3ac6ad-7d53-46fb-a642-24789211e0d7","Type":"ContainerStarted","Data":"e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592"}
Aug 01 01:21:47.049345 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:47.049242    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" event={"ID":"7a3ac6ad-7d53-46fb-a642-24789211e0d7","Type":"ContainerStarted","Data":"2d61fd386ecd5fda220c6d0333e5309798b9e824b5f50dce1e452e3af2e3015d"}
Aug 01 01:21:47.068600 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:47.068514    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" podStartSLOduration=1.068497057 podStartE2EDuration="1.068497057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:21:47.067053598 +0000 UTC m=+2873.300985863" watchObservedRunningTime="2026-08-01 01:21:47.068497057 +0000 UTC m=+2873.302429325"
Aug 01 01:21:47.149141 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:47.149105    2570 patch_prober.go:28] interesting pod/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Aug 01 01:21:47.149282 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:47.149167    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 01:21:47.818370 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:47.818337    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:48.823440 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:48.823409    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:49.060159 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:49.060126    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:49.061141 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:49.061123    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:21:50.089798 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.089764    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"]
Aug 01 01:21:50.090306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.090199    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ca871b-2b99-4ec3-9cdb-a9c86b8e3ed0" containerName="istio-proxy"
Aug 01 01:21:50.093407 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.093383    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.095975 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.095957    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-041b9f-serving-cert\""
Aug 01 01:21:50.106044 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.106026    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"]
Aug 01 01:21:50.189311 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.189280    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8be48a00-2328-4e73-93d9-58df92990c24-maas-api-tls\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.189311 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.189314    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7b9\" (UniqueName: \"kubernetes.io/projected/8be48a00-2328-4e73-93d9-58df92990c24-kube-api-access-vx7b9\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.189629 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.189601    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be48a00-2328-4e73-93d9-58df92990c24-openshift-service-ca\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.290838 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.290803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be48a00-2328-4e73-93d9-58df92990c24-openshift-service-ca\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.290838 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.290844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8be48a00-2328-4e73-93d9-58df92990c24-maas-api-tls\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.291129 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.290861    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7b9\" (UniqueName: \"kubernetes.io/projected/8be48a00-2328-4e73-93d9-58df92990c24-kube-api-access-vx7b9\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.291542 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.291516    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be48a00-2328-4e73-93d9-58df92990c24-openshift-service-ca\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.293429 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.293404    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8be48a00-2328-4e73-93d9-58df92990c24-maas-api-tls\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.298541 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.298520    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7b9\" (UniqueName: \"kubernetes.io/projected/8be48a00-2328-4e73-93d9-58df92990c24-kube-api-access-vx7b9\") pod \"maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.405147 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.405066    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:50.713595 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.713507    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"]
Aug 01 01:21:50.717105 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.717084    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:50.719845 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.719823    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-041b9f\""
Aug 01 01:21:50.720174 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.720158    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-041b9f-dockercfg-4948w\""
Aug 01 01:21:50.730976 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.730942    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"]
Aug 01 01:21:50.746755 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.746727    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"]
Aug 01 01:21:50.750242 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:21:50.750214    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be48a00_2328_4e73_93d9_58df92990c24.slice/crio-1b398e22689e200b2a0db5a201efb5840ad99e32d3e458eef15e54b251fa42fb WatchSource:0}: Error finding container 1b398e22689e200b2a0db5a201efb5840ad99e32d3e458eef15e54b251fa42fb: Status 404 returned error can't find the container with id 1b398e22689e200b2a0db5a201efb5840ad99e32d3e458eef15e54b251fa42fb
Aug 01 01:21:50.752200 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.752182    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 01:21:50.794504 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.794478    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133ac9f7-ec14-41ed-a8ba-83097df00a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq\" (UID: \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:50.794619 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.794588    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7fd\" (UniqueName: \"kubernetes.io/projected/133ac9f7-ec14-41ed-a8ba-83097df00a0b-kube-api-access-6z7fd\") pod \"payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq\" (UID: \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:50.866242 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.866211    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:21:50.895615 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.895590    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-envoy\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.895784 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.895629    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-token\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.895784 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.895713    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-data\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.895922 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.895831    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxgf9\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-kube-api-access-sxgf9\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.895922 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.895885    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-certs\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.895922 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.895920    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istiod-ca-cert\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.896179 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.895963    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-podinfo\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.896179 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896018    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-credential-socket\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.896179 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896059    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-socket\") pod \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\" (UID: \"30eedf06-f7aa-4935-8ba5-e54dc7310d12\") "
Aug 01 01:21:50.896179 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.895961    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-data" (OuterVolumeSpecName: "istio-data") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:50.896179 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896091    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:50.896442 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896212    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:21:50.896442 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896230    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7fd\" (UniqueName: \"kubernetes.io/projected/133ac9f7-ec14-41ed-a8ba-83097df00a0b-kube-api-access-6z7fd\") pod \"payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq\" (UID: \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:50.896442 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896289    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133ac9f7-ec14-41ed-a8ba-83097df00a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq\" (UID: \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:50.896442 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896372    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:50.896650 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896484    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:50.896650 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896510    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:50.896650 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896525    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:50.896650 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896538    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:50.896650 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.896560    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:50.897279 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.897252    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133ac9f7-ec14-41ed-a8ba-83097df00a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq\" (UID: \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:50.898259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.898235    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-token" (OuterVolumeSpecName: "istio-token") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:50.898376 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.898367    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:21:50.898461 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.898428    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-kube-api-access-sxgf9" (OuterVolumeSpecName: "kube-api-access-sxgf9") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "kube-api-access-sxgf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:21:50.898709 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.898687    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30eedf06-f7aa-4935-8ba5-e54dc7310d12" (UID: "30eedf06-f7aa-4935-8ba5-e54dc7310d12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:21:50.903715 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.903693    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7fd\" (UniqueName: \"kubernetes.io/projected/133ac9f7-ec14-41ed-a8ba-83097df00a0b-kube-api-access-6z7fd\") pod \"payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq\" (UID: \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:50.997162 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.997096    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgf9\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-kube-api-access-sxgf9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:50.997162 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.997120    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:50.997162 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.997129    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:50.997162 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.997149    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:50.997162 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:50.997159    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30eedf06-f7aa-4935-8ba5-e54dc7310d12-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:21:51.028989 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.028966    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:51.073764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.073730    2570 generic.go:356] "Generic (PLEG): container finished" podID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerID="37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2" exitCode=0
Aug 01 01:21:51.073962 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.073785    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"
Aug 01 01:21:51.073962 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.073814    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" event={"ID":"30eedf06-f7aa-4935-8ba5-e54dc7310d12","Type":"ContainerDied","Data":"37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2"}
Aug 01 01:21:51.073962 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.073875    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5" event={"ID":"30eedf06-f7aa-4935-8ba5-e54dc7310d12","Type":"ContainerDied","Data":"989f5f0d6933e14de8f39313d1b2691b3f18ababe9bae75da531ed5bac04d258"}
Aug 01 01:21:51.073962 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.073923    2570 scope.go:117] "RemoveContainer" containerID="37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2"
Aug 01 01:21:51.075628 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.075597    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66" event={"ID":"8be48a00-2328-4e73-93d9-58df92990c24","Type":"ContainerStarted","Data":"1b398e22689e200b2a0db5a201efb5840ad99e32d3e458eef15e54b251fa42fb"}
Aug 01 01:21:51.086638 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.086601    2570 scope.go:117] "RemoveContainer" containerID="dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496"
Aug 01 01:21:51.098029 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.098009    2570 scope.go:117] "RemoveContainer" containerID="37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2"
Aug 01 01:21:51.098400 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:51.098346    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2\": container with ID starting with 37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2 not found: ID does not exist" containerID="37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2"
Aug 01 01:21:51.098400 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.098380    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2"} err="failed to get container status \"37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2\": rpc error: code = NotFound desc = could not find container \"37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2\": container with ID starting with 37a219bf87e3013d8f1a9ebf8eebbe0328c63b14aa0efa92ce8787833dd926c2 not found: ID does not exist"
Aug 01 01:21:51.098523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.098403    2570 scope.go:117] "RemoveContainer" containerID="dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496"
Aug 01 01:21:51.098706 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:21:51.098687    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496\": container with ID starting with dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496 not found: ID does not exist" containerID="dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496"
Aug 01 01:21:51.098842 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.098714    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496"} err="failed to get container status \"dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496\": rpc error: code = NotFound desc = could not find container \"dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496\": container with ID starting with dcab5ebeb45d0bcab36c3b1f655942e8085595c25269c4408bf4b05fa88ac496 not found: ID does not exist"
Aug 01 01:21:51.099850 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.099830    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"]
Aug 01 01:21:51.102679 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.102634    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1cf076-openshift-default-f86555bb-x8ml5"]
Aug 01 01:21:51.167404 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:51.167369    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"]
Aug 01 01:21:51.173517 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:21:51.173492    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133ac9f7_ec14_41ed_a8ba_83097df00a0b.slice/crio-cac1b8949faab865797fc5df80f048d6f401df231998d5a0aa4a5e93f1a02abd WatchSource:0}: Error finding container cac1b8949faab865797fc5df80f048d6f401df231998d5a0aa4a5e93f1a02abd: Status 404 returned error can't find the container with id cac1b8949faab865797fc5df80f048d6f401df231998d5a0aa4a5e93f1a02abd
Aug 01 01:21:52.082846 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:52.082808    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq" event={"ID":"133ac9f7-ec14-41ed-a8ba-83097df00a0b","Type":"ContainerStarted","Data":"fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6"}
Aug 01 01:21:52.083062 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:52.082865    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq" event={"ID":"133ac9f7-ec14-41ed-a8ba-83097df00a0b","Type":"ContainerStarted","Data":"cac1b8949faab865797fc5df80f048d6f401df231998d5a0aa4a5e93f1a02abd"}
Aug 01 01:21:52.083062 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:52.082904    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:21:52.085209 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:52.085181    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66" event={"ID":"8be48a00-2328-4e73-93d9-58df92990c24","Type":"ContainerStarted","Data":"bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07"}
Aug 01 01:21:52.085356 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:52.085319    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:21:52.098910 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:52.098866    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq" podStartSLOduration=2.09885205 podStartE2EDuration="2.09885205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:21:52.097419619 +0000 UTC m=+2878.331351889" watchObservedRunningTime="2026-08-01 01:21:52.09885205 +0000 UTC m=+2878.332784306"
Aug 01 01:21:52.113976 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:52.113921    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66" podStartSLOduration=1.69041212 podStartE2EDuration="2.113907636s" totalImagesPullingTime="423.495516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:21:52.112086659 +0000 UTC m=+2878.346018928" watchObservedRunningTime="2026-08-01 01:21:52.113907636 +0000 UTC m=+2878.347839938"
Aug 01 01:21:52.263565 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:52.263525    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" path="/var/lib/kubelet/pods/30eedf06-f7aa-4935-8ba5-e54dc7310d12/volumes"
Aug 01 01:21:58.096355 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:21:58.096322    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:22:03.093107 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:03.093072    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:22:14.844526 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.844487    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"]
Aug 01 01:22:14.844965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.844885    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy"
Aug 01 01:22:14.844965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.844903    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy"
Aug 01 01:22:14.844965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.844925    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy"
Aug 01 01:22:14.844965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.844948    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy"
Aug 01 01:22:14.845174 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.845028    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy"
Aug 01 01:22:14.845174 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.845043    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="30eedf06-f7aa-4935-8ba5-e54dc7310d12" containerName="istio-proxy"
Aug 01 01:22:14.847764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.847745    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:14.850163 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.850143    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a9f952-openshift-default-dockercfg-fwxbz\""
Aug 01 01:22:14.858178 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:14.858152    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"]
Aug 01 01:22:15.007299 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007264    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istiod-ca-cert\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.007299 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007308    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-envoy\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.007528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007335    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-data\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.007528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007414    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-token\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.007528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007453    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-certs\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.007528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007484    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-socket\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.007528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007517    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-podinfo\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.007694 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007539    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-credential-socket\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.007694 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.007563    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dqr\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-kube-api-access-r7dqr\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.108894 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.108800    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istiod-ca-cert\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.108894 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.108843    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-envoy\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.108894 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.108875    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-data\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109203 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.108918    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-token\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109203 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.108981    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-certs\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109203 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-socket\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109203 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109164    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-podinfo\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-credential-socket\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109245    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dqr\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-kube-api-access-r7dqr\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109347    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-socket\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109410    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-data\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109451    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-certs\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109716 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109595    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-credential-socket\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.109716 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.109637    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istiod-ca-cert\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.111289 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.111269    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-envoy\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.111506 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.111485    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-podinfo\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.116872 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.116850    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-token\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.117022 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.116922    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dqr\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-kube-api-access-r7dqr\") pod \"e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") " pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.159696 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.159672    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:15.291749 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.291718    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"]
Aug 01 01:22:15.294232 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:22:15.294208    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2d08dd_8d94_4d17_90f5_ab156c642b1e.slice/crio-be2df7776ee65b225576c29d263e584957c09f99960c0fa748fba8b3e03025d1 WatchSource:0}: Error finding container be2df7776ee65b225576c29d263e584957c09f99960c0fa748fba8b3e03025d1: Status 404 returned error can't find the container with id be2df7776ee65b225576c29d263e584957c09f99960c0fa748fba8b3e03025d1
Aug 01 01:22:15.296460 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.296426    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:15.296563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.296545    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:15.296638 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:15.296593    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:16.200061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:16.200020    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" event={"ID":"bb2d08dd-8d94-4d17-90f5-ab156c642b1e","Type":"ContainerStarted","Data":"5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69"}
Aug 01 01:22:16.200463 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:16.200076    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" event={"ID":"bb2d08dd-8d94-4d17-90f5-ab156c642b1e","Type":"ContainerStarted","Data":"be2df7776ee65b225576c29d263e584957c09f99960c0fa748fba8b3e03025d1"}
Aug 01 01:22:16.220492 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:16.220444    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" podStartSLOduration=2.2204295800000002 podStartE2EDuration="2.22042958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:22:16.219580122 +0000 UTC m=+2902.453512388" watchObservedRunningTime="2026-08-01 01:22:16.22042958 +0000 UTC m=+2902.454361846"
Aug 01 01:22:17.160633 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:17.160599    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:17.166590 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:17.166562    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:17.205293 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:17.205268    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:17.206418 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:17.206391    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:19.342070 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.342033    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"]
Aug 01 01:22:19.346441 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.346420    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:19.348896 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.348875    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a9f952-serving-cert\""
Aug 01 01:22:19.362847 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.361767    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"]
Aug 01 01:22:19.463390 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.463352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw89w\" (UniqueName: \"kubernetes.io/projected/ced844a3-9136-4c30-b905-87775fceee57-kube-api-access-fw89w\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:19.463709 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.463680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced844a3-9136-4c30-b905-87775fceee57-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:19.463909 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.463890    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:19.565207 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.565168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw89w\" (UniqueName: \"kubernetes.io/projected/ced844a3-9136-4c30-b905-87775fceee57-kube-api-access-fw89w\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:19.565521 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.565500    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced844a3-9136-4c30-b905-87775fceee57-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:19.566552 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.566527    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:19.566757 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.566426    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced844a3-9136-4c30-b905-87775fceee57-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:19.566884 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:22:19.566759    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-serving-cert: secret "maas-api-e2e-api-b-a9f952-serving-cert" not found
Aug 01 01:22:19.567055 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:22:19.567043    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls podName:ced844a3-9136-4c30-b905-87775fceee57 nodeName:}" failed. No retries permitted until 2026-08-01 01:22:20.067020684 +0000 UTC m=+2906.300952943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls") pod "maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc" (UID: "ced844a3-9136-4c30-b905-87775fceee57") : secret "maas-api-e2e-api-b-a9f952-serving-cert" not found
Aug 01 01:22:19.575271 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:19.575243    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw89w\" (UniqueName: \"kubernetes.io/projected/ced844a3-9136-4c30-b905-87775fceee57-kube-api-access-fw89w\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:20.072491 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.072450    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:20.076030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.076004    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls\") pod \"maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:20.159725 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.159681    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"]
Aug 01 01:22:20.166563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.166536    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:20.171145 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.171120    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a9f952-dockercfg-6vkq2\""
Aug 01 01:22:20.171474 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.171202    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a9f952\""
Aug 01 01:22:20.179640 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.179617    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"]
Aug 01 01:22:20.269832 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.269796    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:20.274209 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.274169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk\" (UID: \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:20.274371 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.274253    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrj5\" (UniqueName: \"kubernetes.io/projected/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-kube-api-access-kmrj5\") pod \"payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk\" (UID: \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:20.379068 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.375378    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk\" (UID: \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:20.379068 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.375469    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrj5\" (UniqueName: \"kubernetes.io/projected/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-kube-api-access-kmrj5\") pod \"payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk\" (UID: \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:20.384998 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.384290    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk\" (UID: \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:20.387290 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.387237    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrj5\" (UniqueName: \"kubernetes.io/projected/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-kube-api-access-kmrj5\") pod \"payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk\" (UID: \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:20.435046 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.435015    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"]
Aug 01 01:22:20.437015 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:22:20.436987    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced844a3_9136_4c30_b905_87775fceee57.slice/crio-185d13c0f85d8222ee08cdfd205a213fef3332f22e77b52b6edda09278a1c368 WatchSource:0}: Error finding container 185d13c0f85d8222ee08cdfd205a213fef3332f22e77b52b6edda09278a1c368: Status 404 returned error can't find the container with id 185d13c0f85d8222ee08cdfd205a213fef3332f22e77b52b6edda09278a1c368
Aug 01 01:22:20.483245 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.483216    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:20.847172 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:20.847147    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"]
Aug 01 01:22:20.849471 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:22:20.849439    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4bfe5d_fbe2_40d3_ad1b_ad76d174383a.slice/crio-16e6968e3c5b840c3aeb6013956aa7db1a0cd9098aae5d59984f83e0b53e6bf6 WatchSource:0}: Error finding container 16e6968e3c5b840c3aeb6013956aa7db1a0cd9098aae5d59984f83e0b53e6bf6: Status 404 returned error can't find the container with id 16e6968e3c5b840c3aeb6013956aa7db1a0cd9098aae5d59984f83e0b53e6bf6
Aug 01 01:22:21.228453 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:21.228355    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk" event={"ID":"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a","Type":"ContainerStarted","Data":"a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598"}
Aug 01 01:22:21.228453 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:21.228422    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk" event={"ID":"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a","Type":"ContainerStarted","Data":"16e6968e3c5b840c3aeb6013956aa7db1a0cd9098aae5d59984f83e0b53e6bf6"}
Aug 01 01:22:21.228713 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:21.228551    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:21.230061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:21.230036    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc" event={"ID":"ced844a3-9136-4c30-b905-87775fceee57","Type":"ContainerStarted","Data":"5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2"}
Aug 01 01:22:21.230165 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:21.230092    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:21.230165 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:21.230117    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc" event={"ID":"ced844a3-9136-4c30-b905-87775fceee57","Type":"ContainerStarted","Data":"185d13c0f85d8222ee08cdfd205a213fef3332f22e77b52b6edda09278a1c368"}
Aug 01 01:22:21.244259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:21.244216    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk" podStartSLOduration=1.244202615 podStartE2EDuration="1.244202615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:22:21.243728054 +0000 UTC m=+2907.477660321" watchObservedRunningTime="2026-08-01 01:22:21.244202615 +0000 UTC m=+2907.478134880"
Aug 01 01:22:21.260727 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:21.260668    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc" podStartSLOduration=1.913274616 podStartE2EDuration="2.26064873s" totalImagesPullingTime="347.374114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:22:21.258196013 +0000 UTC m=+2907.492128280" watchObservedRunningTime="2026-08-01 01:22:21.26064873 +0000 UTC m=+2907.494580998"
Aug 01 01:22:27.241132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:27.241102    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:22:27.270019 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:27.269979    2570 generic.go:356] "Generic (PLEG): container finished" podID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerID="e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592" exitCode=0
Aug 01 01:22:27.270206 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:27.270093    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" event={"ID":"7a3ac6ad-7d53-46fb-a642-24789211e0d7","Type":"ContainerDied","Data":"e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592"}
Aug 01 01:22:27.270581 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:27.270555    2570 scope.go:117] "RemoveContainer" containerID="e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592"
Aug 01 01:22:27.272109 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:27.272073    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:27.272212 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:27.272157    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:27.272212 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:27.272205    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:28.280047 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:28.280002    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" event={"ID":"7a3ac6ad-7d53-46fb-a642-24789211e0d7","Type":"ContainerStarted","Data":"1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359"}
Aug 01 01:22:28.819108 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:28.819073    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:22:28.819298 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:28.819275    2570 patch_prober.go:28] interesting pod/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Aug 01 01:22:28.819360 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:28.819340    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Aug 01 01:22:29.819477 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:29.819434    2570 patch_prober.go:28] interesting pod/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Aug 01 01:22:29.819988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:29.819510    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Aug 01 01:22:30.819083 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:30.819046    2570 patch_prober.go:28] interesting pod/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Aug 01 01:22:30.819252 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:30.819104    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Aug 01 01:22:31.819267 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:31.819234    2570 patch_prober.go:28] interesting pod/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Aug 01 01:22:31.819689 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:31.819288    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Aug 01 01:22:32.237677 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:32.237587    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:22:32.605405 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:32.605381    2570 patch_prober.go:28] interesting pod/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": read tcp 10.133.0.2:54148->10.133.0.79:15021: read: connection reset by peer" start-of-body=
Aug 01 01:22:32.605594 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:32.605573    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": read tcp 10.133.0.2:54148->10.133.0.79:15021: read: connection reset by peer"
Aug 01 01:22:32.818840 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:32.818809    2570 patch_prober.go:28] interesting pod/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Aug 01 01:22:32.819016 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:32.818874    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Aug 01 01:22:33.310359 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.310328    2570 generic.go:356] "Generic (PLEG): container finished" podID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerID="5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69" exitCode=0
Aug 01 01:22:33.310748 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.310412    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" event={"ID":"bb2d08dd-8d94-4d17-90f5-ab156c642b1e","Type":"ContainerDied","Data":"5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69"}
Aug 01 01:22:33.310820 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.310803    2570 scope.go:117] "RemoveContainer" containerID="5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69"
Aug 01 01:22:33.311830 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.311802    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:33.311883 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.311872    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:33.311919 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.311904    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:22:33.823506 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.823469    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:22:33.823877 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.823855    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:22:33.824436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:33.824418    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:22:34.317418 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:34.317369    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" event={"ID":"bb2d08dd-8d94-4d17-90f5-ab156c642b1e","Type":"ContainerStarted","Data":"ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338"}
Aug 01 01:22:35.160842 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:35.160806    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:35.161283 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:35.161195    2570 patch_prober.go:28] interesting pod/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Aug 01 01:22:35.161479 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:35.161456    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Aug 01 01:22:36.160530 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:36.160490    2570 patch_prober.go:28] interesting pod/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Aug 01 01:22:36.161129 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:36.161096    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Aug 01 01:22:37.160574 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:37.160527    2570 patch_prober.go:28] interesting pod/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Aug 01 01:22:37.161075 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:37.160631    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Aug 01 01:22:38.161223 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:38.161179    2570 patch_prober.go:28] interesting pod/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Aug 01 01:22:38.161858 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:38.161822    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Aug 01 01:22:39.160481 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:39.160431    2570 patch_prober.go:28] interesting pod/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Aug 01 01:22:39.160699 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:39.160516    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Aug 01 01:22:40.164390 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:40.164356    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:40.164896 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:40.164732    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:22:40.165360 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:22:40.165338    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:23:01.661445 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:01.661412    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"]
Aug 01 01:23:01.662107 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:01.661670    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66" podUID="8be48a00-2328-4e73-93d9-58df92990c24" containerName="maas-api" containerID="cri-o://bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07" gracePeriod=30
Aug 01 01:23:01.721562 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:01.721526    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"]
Aug 01 01:23:01.721795 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:01.721754    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq" podUID="133ac9f7-ec14-41ed-a8ba-83097df00a0b" containerName="payload-pre-processing" containerID="cri-o://fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6" gracePeriod=30
Aug 01 01:23:01.972223 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:01.972197    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:23:02.003494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.003470    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:23:02.053203 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.053176    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8be48a00-2328-4e73-93d9-58df92990c24-maas-api-tls\") pod \"8be48a00-2328-4e73-93d9-58df92990c24\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") "
Aug 01 01:23:02.053388 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.053215    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx7b9\" (UniqueName: \"kubernetes.io/projected/8be48a00-2328-4e73-93d9-58df92990c24-kube-api-access-vx7b9\") pod \"8be48a00-2328-4e73-93d9-58df92990c24\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") "
Aug 01 01:23:02.053467 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.053418    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be48a00-2328-4e73-93d9-58df92990c24-openshift-service-ca\") pod \"8be48a00-2328-4e73-93d9-58df92990c24\" (UID: \"8be48a00-2328-4e73-93d9-58df92990c24\") "
Aug 01 01:23:02.053742 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.053715    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be48a00-2328-4e73-93d9-58df92990c24-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8be48a00-2328-4e73-93d9-58df92990c24" (UID: "8be48a00-2328-4e73-93d9-58df92990c24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:23:02.055184 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.055166    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be48a00-2328-4e73-93d9-58df92990c24-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8be48a00-2328-4e73-93d9-58df92990c24" (UID: "8be48a00-2328-4e73-93d9-58df92990c24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:23:02.055266 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.055233    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be48a00-2328-4e73-93d9-58df92990c24-kube-api-access-vx7b9" (OuterVolumeSpecName: "kube-api-access-vx7b9") pod "8be48a00-2328-4e73-93d9-58df92990c24" (UID: "8be48a00-2328-4e73-93d9-58df92990c24"). InnerVolumeSpecName "kube-api-access-vx7b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:23:02.154981 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.154947    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z7fd\" (UniqueName: \"kubernetes.io/projected/133ac9f7-ec14-41ed-a8ba-83097df00a0b-kube-api-access-6z7fd\") pod \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\" (UID: \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\") "
Aug 01 01:23:02.155159 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.155047    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133ac9f7-ec14-41ed-a8ba-83097df00a0b-plugins-config-volume\") pod \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\" (UID: \"133ac9f7-ec14-41ed-a8ba-83097df00a0b\") "
Aug 01 01:23:02.155230 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.155195    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be48a00-2328-4e73-93d9-58df92990c24-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:02.155230 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.155206    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8be48a00-2328-4e73-93d9-58df92990c24-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:02.155230 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.155217    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx7b9\" (UniqueName: \"kubernetes.io/projected/8be48a00-2328-4e73-93d9-58df92990c24-kube-api-access-vx7b9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:02.155469 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.155443    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133ac9f7-ec14-41ed-a8ba-83097df00a0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "133ac9f7-ec14-41ed-a8ba-83097df00a0b" (UID: "133ac9f7-ec14-41ed-a8ba-83097df00a0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:23:02.157001 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.156979    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133ac9f7-ec14-41ed-a8ba-83097df00a0b-kube-api-access-6z7fd" (OuterVolumeSpecName: "kube-api-access-6z7fd") pod "133ac9f7-ec14-41ed-a8ba-83097df00a0b" (UID: "133ac9f7-ec14-41ed-a8ba-83097df00a0b"). InnerVolumeSpecName "kube-api-access-6z7fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:23:02.256542 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.256449    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133ac9f7-ec14-41ed-a8ba-83097df00a0b-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:02.256542 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.256506    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7fd\" (UniqueName: \"kubernetes.io/projected/133ac9f7-ec14-41ed-a8ba-83097df00a0b-kube-api-access-6z7fd\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:02.455862 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.455828    2570 generic.go:356] "Generic (PLEG): container finished" podID="133ac9f7-ec14-41ed-a8ba-83097df00a0b" containerID="fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6" exitCode=0
Aug 01 01:23:02.456066 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.455868    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"
Aug 01 01:23:02.456066 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.455911    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq" event={"ID":"133ac9f7-ec14-41ed-a8ba-83097df00a0b","Type":"ContainerDied","Data":"fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6"}
Aug 01 01:23:02.456066 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.455985    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq" event={"ID":"133ac9f7-ec14-41ed-a8ba-83097df00a0b","Type":"ContainerDied","Data":"cac1b8949faab865797fc5df80f048d6f401df231998d5a0aa4a5e93f1a02abd"}
Aug 01 01:23:02.456066 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.456015    2570 scope.go:117] "RemoveContainer" containerID="fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6"
Aug 01 01:23:02.457295 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.457268    2570 generic.go:356] "Generic (PLEG): container finished" podID="8be48a00-2328-4e73-93d9-58df92990c24" containerID="bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07" exitCode=0
Aug 01 01:23:02.457433 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.457319    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66" event={"ID":"8be48a00-2328-4e73-93d9-58df92990c24","Type":"ContainerDied","Data":"bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07"}
Aug 01 01:23:02.457433 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.457356    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66" event={"ID":"8be48a00-2328-4e73-93d9-58df92990c24","Type":"ContainerDied","Data":"1b398e22689e200b2a0db5a201efb5840ad99e32d3e458eef15e54b251fa42fb"}
Aug 01 01:23:02.457433 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.457321    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"
Aug 01 01:23:02.465062 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.465043    2570 scope.go:117] "RemoveContainer" containerID="fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6"
Aug 01 01:23:02.465319 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:23:02.465299    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6\": container with ID starting with fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6 not found: ID does not exist" containerID="fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6"
Aug 01 01:23:02.465394 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.465325    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6"} err="failed to get container status \"fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6\": rpc error: code = NotFound desc = could not find container \"fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6\": container with ID starting with fac5af1d155c377038f6dad30c0938ceb7dcadb385611bfc36080220e9cb35f6 not found: ID does not exist"
Aug 01 01:23:02.465394 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.465340    2570 scope.go:117] "RemoveContainer" containerID="bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07"
Aug 01 01:23:02.472640 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.472618    2570 scope.go:117] "RemoveContainer" containerID="bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07"
Aug 01 01:23:02.472919 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:23:02.472883    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07\": container with ID starting with bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07 not found: ID does not exist" containerID="bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07"
Aug 01 01:23:02.473067 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.472915    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07"} err="failed to get container status \"bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07\": rpc error: code = NotFound desc = could not find container \"bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07\": container with ID starting with bdf04a8ec3d62101e6394c6da5fa03511c7668f8f148c3d07fbcd5e53d1f0f07 not found: ID does not exist"
Aug 01 01:23:02.475404 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.475382    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"]
Aug 01 01:23:02.478604 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.478583    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-041b9f-86b6d568c5-hxp66"]
Aug 01 01:23:02.488650 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.488625    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"]
Aug 01 01:23:02.491960 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:02.491924    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-041b9f-65c67f74bd-2zxhq"]
Aug 01 01:23:04.269185 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:04.269151    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133ac9f7-ec14-41ed-a8ba-83097df00a0b" path="/var/lib/kubelet/pods/133ac9f7-ec14-41ed-a8ba-83097df00a0b/volumes"
Aug 01 01:23:04.269567 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:04.269550    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be48a00-2328-4e73-93d9-58df92990c24" path="/var/lib/kubelet/pods/8be48a00-2328-4e73-93d9-58df92990c24/volumes"
Aug 01 01:23:08.156872 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:08.156836    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"]
Aug 01 01:23:08.157349 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:08.157119    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy" containerID="cri-o://1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359" gracePeriod=30
Aug 01 01:23:13.404296 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.404268    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:23:13.519167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.519082    2570 generic.go:356] "Generic (PLEG): container finished" podID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerID="1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359" exitCode=0
Aug 01 01:23:13.519167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.519124    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" event={"ID":"7a3ac6ad-7d53-46fb-a642-24789211e0d7","Type":"ContainerDied","Data":"1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359"}
Aug 01 01:23:13.519167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.519131    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"
Aug 01 01:23:13.519427 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.519177    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn" event={"ID":"7a3ac6ad-7d53-46fb-a642-24789211e0d7","Type":"ContainerDied","Data":"2d61fd386ecd5fda220c6d0333e5309798b9e824b5f50dce1e452e3af2e3015d"}
Aug 01 01:23:13.519427 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.519207    2570 scope.go:117] "RemoveContainer" containerID="1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359"
Aug 01 01:23:13.527776 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.527758    2570 scope.go:117] "RemoveContainer" containerID="e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592"
Aug 01 01:23:13.535228 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.535209    2570 scope.go:117] "RemoveContainer" containerID="1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359"
Aug 01 01:23:13.535473 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:23:13.535457    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359\": container with ID starting with 1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359 not found: ID does not exist" containerID="1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359"
Aug 01 01:23:13.535520 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.535479    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359"} err="failed to get container status \"1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359\": rpc error: code = NotFound desc = could not find container \"1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359\": container with ID starting with 1f3c2ed214b61376d7dff10be0acac82c92432f0127133e692fa036e613bf359 not found: ID does not exist"
Aug 01 01:23:13.535520 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.535495    2570 scope.go:117] "RemoveContainer" containerID="e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592"
Aug 01 01:23:13.535733 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:23:13.535714    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592\": container with ID starting with e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592 not found: ID does not exist" containerID="e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592"
Aug 01 01:23:13.535781 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.535741    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592"} err="failed to get container status \"e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592\": rpc error: code = NotFound desc = could not find container \"e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592\": container with ID starting with e9eaff501c49973da0fa3ed9afd115857519a6056cac759c3bcd0a300f4bf592 not found: ID does not exist"
Aug 01 01:23:13.547101 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547080    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-socket\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547170 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547121    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-certs\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547170 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547138    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-token\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547170 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547163    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-data\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547271 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547224    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-credential-socket\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547271 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547247    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-podinfo\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547287    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-envoy\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547311    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-927fn\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-kube-api-access-927fn\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547335    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istiod-ca-cert\") pod \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\" (UID: \"7a3ac6ad-7d53-46fb-a642-24789211e0d7\") "
Aug 01 01:23:13.547502 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547369    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:13.547502 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547389    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:13.547611 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547502    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:13.547691 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547661    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.547754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547692    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.547754 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547709    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.547910 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547885    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:13.547985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.547957    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:23:13.549326 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.549297    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-kube-api-access-927fn" (OuterVolumeSpecName: "kube-api-access-927fn") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "kube-api-access-927fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:23:13.549326 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.549309    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:23:13.549577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.549563    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:23:13.549655 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.549633    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a3ac6ad-7d53-46fb-a642-24789211e0d7" (UID: "7a3ac6ad-7d53-46fb-a642-24789211e0d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:13.648727 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.648705    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.648727 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.648729    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.648856 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.648738    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.648856 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.648747    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927fn\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-kube-api-access-927fn\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.648856 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.648756    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.648856 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.648771    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3ac6ad-7d53-46fb-a642-24789211e0d7-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:13.840630 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.840605    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"]
Aug 01 01:23:13.844398 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:13.844367    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-041b9f-openshift-default-749845db84-fgrbn"]
Aug 01 01:23:14.263494 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:14.263414    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" path="/var/lib/kubelet/pods/7a3ac6ad-7d53-46fb-a642-24789211e0d7/volumes"
Aug 01 01:23:24.092969 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.092901    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"]
Aug 01 01:23:24.093473 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.093236    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc" podUID="ced844a3-9136-4c30-b905-87775fceee57" containerName="maas-api" containerID="cri-o://5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2" gracePeriod=30
Aug 01 01:23:24.152851 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.152821    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"]
Aug 01 01:23:24.153132 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.153080    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk" podUID="fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a" containerName="payload-processing" containerID="cri-o://a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598" gracePeriod=30
Aug 01 01:23:24.398657 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.398633    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:23:24.423332 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.423307    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:23:24.432920 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.432900    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmrj5\" (UniqueName: \"kubernetes.io/projected/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-kube-api-access-kmrj5\") pod \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\" (UID: \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\") "
Aug 01 01:23:24.433064 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.432960    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls\") pod \"ced844a3-9136-4c30-b905-87775fceee57\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") "
Aug 01 01:23:24.433064 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.433013    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-plugins-config-volume\") pod \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\" (UID: \"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a\") "
Aug 01 01:23:24.433186 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.433133    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced844a3-9136-4c30-b905-87775fceee57-openshift-service-ca\") pod \"ced844a3-9136-4c30-b905-87775fceee57\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") "
Aug 01 01:23:24.433186 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.433175    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw89w\" (UniqueName: \"kubernetes.io/projected/ced844a3-9136-4c30-b905-87775fceee57-kube-api-access-fw89w\") pod \"ced844a3-9136-4c30-b905-87775fceee57\" (UID: \"ced844a3-9136-4c30-b905-87775fceee57\") "
Aug 01 01:23:24.433433 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.433403    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a" (UID: "fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:23:24.433548 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.433473    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced844a3-9136-4c30-b905-87775fceee57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ced844a3-9136-4c30-b905-87775fceee57" (UID: "ced844a3-9136-4c30-b905-87775fceee57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:23:24.434973 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.434907    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-kube-api-access-kmrj5" (OuterVolumeSpecName: "kube-api-access-kmrj5") pod "fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a" (UID: "fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a"). InnerVolumeSpecName "kube-api-access-kmrj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:23:24.435061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.434980    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ced844a3-9136-4c30-b905-87775fceee57" (UID: "ced844a3-9136-4c30-b905-87775fceee57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:23:24.435188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.435164    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced844a3-9136-4c30-b905-87775fceee57-kube-api-access-fw89w" (OuterVolumeSpecName: "kube-api-access-fw89w") pod "ced844a3-9136-4c30-b905-87775fceee57" (UID: "ced844a3-9136-4c30-b905-87775fceee57"). InnerVolumeSpecName "kube-api-access-fw89w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:23:24.534413 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.534388    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:24.534413 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.534412    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced844a3-9136-4c30-b905-87775fceee57-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:24.534413 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.534422    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw89w\" (UniqueName: \"kubernetes.io/projected/ced844a3-9136-4c30-b905-87775fceee57-kube-api-access-fw89w\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:24.534602 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.534430    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrj5\" (UniqueName: \"kubernetes.io/projected/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a-kube-api-access-kmrj5\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:24.534602 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.534438    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ced844a3-9136-4c30-b905-87775fceee57-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:24.579526 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.579499    2570 generic.go:356] "Generic (PLEG): container finished" podID="fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a" containerID="a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598" exitCode=0
Aug 01 01:23:24.579651 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.579541    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"
Aug 01 01:23:24.579651 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.579583    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk" event={"ID":"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a","Type":"ContainerDied","Data":"a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598"}
Aug 01 01:23:24.579651 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.579644    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk" event={"ID":"fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a","Type":"ContainerDied","Data":"16e6968e3c5b840c3aeb6013956aa7db1a0cd9098aae5d59984f83e0b53e6bf6"}
Aug 01 01:23:24.579833 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.579682    2570 scope.go:117] "RemoveContainer" containerID="a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598"
Aug 01 01:23:24.580962 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.580921    2570 generic.go:356] "Generic (PLEG): container finished" podID="ced844a3-9136-4c30-b905-87775fceee57" containerID="5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2" exitCode=0
Aug 01 01:23:24.581093 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.580979    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"
Aug 01 01:23:24.581093 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.581021    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc" event={"ID":"ced844a3-9136-4c30-b905-87775fceee57","Type":"ContainerDied","Data":"5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2"}
Aug 01 01:23:24.581093 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.581073    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc" event={"ID":"ced844a3-9136-4c30-b905-87775fceee57","Type":"ContainerDied","Data":"185d13c0f85d8222ee08cdfd205a213fef3332f22e77b52b6edda09278a1c368"}
Aug 01 01:23:24.589041 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.589006    2570 scope.go:117] "RemoveContainer" containerID="a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598"
Aug 01 01:23:24.589289 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:23:24.589259    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598\": container with ID starting with a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598 not found: ID does not exist" containerID="a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598"
Aug 01 01:23:24.589380 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.589299    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598"} err="failed to get container status \"a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598\": rpc error: code = NotFound desc = could not find container \"a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598\": container with ID starting with a39a6f428dda33033adf4324c14851a4a7407fad692343a7ac389d89e9af2598 not found: ID does not exist"
Aug 01 01:23:24.589380 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.589334    2570 scope.go:117] "RemoveContainer" containerID="5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2"
Aug 01 01:23:24.598833 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.598816    2570 scope.go:117] "RemoveContainer" containerID="5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2"
Aug 01 01:23:24.599138 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:23:24.599116    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2\": container with ID starting with 5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2 not found: ID does not exist" containerID="5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2"
Aug 01 01:23:24.599218 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.599140    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2"} err="failed to get container status \"5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2\": rpc error: code = NotFound desc = could not find container \"5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2\": container with ID starting with 5ee8f23cb9abfb2fa179935c0289809b976d5b8337698741bd1b50b6f90470f2 not found: ID does not exist"
Aug 01 01:23:24.603773 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.603750    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"]
Aug 01 01:23:24.609297 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.609243    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9f952-56496bdd5b-v9xnc"]
Aug 01 01:23:24.618371 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.618347    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"]
Aug 01 01:23:24.621882 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:24.621863    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9f952-78d67fb5fd-2bkfk"]
Aug 01 01:23:26.261386 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:26.261344    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ced844a3-9136-4c30-b905-87775fceee57" path="/var/lib/kubelet/pods/ced844a3-9136-4c30-b905-87775fceee57/volumes"
Aug 01 01:23:26.261898 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:26.261836    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a" path="/var/lib/kubelet/pods/fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a/volumes"
Aug 01 01:23:30.878261 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:30.878215    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"]
Aug 01 01:23:30.878732 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:30.878474    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy" containerID="cri-o://ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338" gracePeriod=30
Aug 01 01:23:31.845069 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845038    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"]
Aug 01 01:23:31.845381 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845368    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133ac9f7-ec14-41ed-a8ba-83097df00a0b" containerName="payload-pre-processing"
Aug 01 01:23:31.845381 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845383    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="133ac9f7-ec14-41ed-a8ba-83097df00a0b" containerName="payload-pre-processing"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845400    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845406    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845414    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a" containerName="payload-processing"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845420    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a" containerName="payload-processing"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845429    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845435    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845446    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be48a00-2328-4e73-93d9-58df92990c24" containerName="maas-api"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845451    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be48a00-2328-4e73-93d9-58df92990c24" containerName="maas-api"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845462    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced844a3-9136-4c30-b905-87775fceee57" containerName="maas-api"
Aug 01 01:23:31.845470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845467    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced844a3-9136-4c30-b905-87775fceee57" containerName="maas-api"
Aug 01 01:23:31.845764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845527    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy"
Aug 01 01:23:31.845764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845534    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a3ac6ad-7d53-46fb-a642-24789211e0d7" containerName="istio-proxy"
Aug 01 01:23:31.845764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845541    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced844a3-9136-4c30-b905-87775fceee57" containerName="maas-api"
Aug 01 01:23:31.845764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845548    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="133ac9f7-ec14-41ed-a8ba-83097df00a0b" containerName="payload-pre-processing"
Aug 01 01:23:31.845764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845554    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be48a00-2328-4e73-93d9-58df92990c24" containerName="maas-api"
Aug 01 01:23:31.845764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.845560    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe4bfe5d-fbe2-40d3-ad1b-ad76d174383a" containerName="payload-processing"
Aug 01 01:23:31.848516 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.848500    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.851001 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.850980    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-423d15-openshift-default-dockercfg-pfv84\""
Aug 01 01:23:31.857844 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.857821    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"]
Aug 01 01:23:31.886235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886211    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-certs\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.886523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886245    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-socket\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.886523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886276    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab752358-b918-4050-98df-3c17df616610-istiod-ca-cert\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.886523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886301    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab752358-b918-4050-98df-3c17df616610-istio-podinfo\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.886523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-credential-socket\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.886523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886385    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q545h\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-kube-api-access-q545h\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.886523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-envoy\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.886523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886482    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-data\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.886523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.886511    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-istio-token\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987320 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987286    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab752358-b918-4050-98df-3c17df616610-istiod-ca-cert\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987320 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987322    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab752358-b918-4050-98df-3c17df616610-istio-podinfo\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987504 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987350    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-credential-socket\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987504 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987438    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q545h\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-kube-api-access-q545h\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987504 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987481    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-envoy\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987504 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987497    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-data\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987709 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987514    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-istio-token\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987816 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987792    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-certs\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987877    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-data\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.987906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987812    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-credential-socket\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.988102 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.987958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-socket\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.988102 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.988030    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-certs\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.988217 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.988152    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab752358-b918-4050-98df-3c17df616610-istiod-ca-cert\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.988260 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.988230    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-socket\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.989768 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.989742    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-envoy\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.989921 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.989903    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab752358-b918-4050-98df-3c17df616610-istio-podinfo\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.995437 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.995414    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-istio-token\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:31.995583 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:31.995562    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q545h\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-kube-api-access-q545h\") pod \"e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") " pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:32.159985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:32.159896    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:32.293209 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:32.293117    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"]
Aug 01 01:23:32.295496 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:23:32.295467    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab752358_b918_4050_98df_3c17df616610.slice/crio-610bb776bbeef5dc89c770a16cdd350375ec9c077fe2525709db399bdaaa96a8 WatchSource:0}: Error finding container 610bb776bbeef5dc89c770a16cdd350375ec9c077fe2525709db399bdaaa96a8: Status 404 returned error can't find the container with id 610bb776bbeef5dc89c770a16cdd350375ec9c077fe2525709db399bdaaa96a8
Aug 01 01:23:32.297500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:32.297463    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:23:32.297578 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:32.297550    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:23:32.297623 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:32.297602    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:23:32.619745 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:32.619711    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" event={"ID":"ab752358-b918-4050-98df-3c17df616610","Type":"ContainerStarted","Data":"303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299"}
Aug 01 01:23:32.619903 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:32.619764    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" event={"ID":"ab752358-b918-4050-98df-3c17df616610","Type":"ContainerStarted","Data":"610bb776bbeef5dc89c770a16cdd350375ec9c077fe2525709db399bdaaa96a8"}
Aug 01 01:23:32.638389 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:32.638343    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" podStartSLOduration=1.638328378 podStartE2EDuration="1.638328378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:23:32.636384341 +0000 UTC m=+2978.870316619" watchObservedRunningTime="2026-08-01 01:23:32.638328378 +0000 UTC m=+2978.872260641"
Aug 01 01:23:33.160785 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:33.160759    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:33.165427 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:33.165399    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:33.627061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:33.627028    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:33.630271 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:33.630242    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:36.123721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.123697    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:23:36.221286 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221216    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-socket\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221286 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221251    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7dqr\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-kube-api-access-r7dqr\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221294    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-certs\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221316    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istiod-ca-cert\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221333    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-credential-socket\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221358    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-podinfo\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221423    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-token\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221461    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-data\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221488    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-envoy\") pod \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\" (UID: \"bb2d08dd-8d94-4d17-90f5-ab156c642b1e\") "
Aug 01 01:23:36.221500 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221490    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:36.221906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221619    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:36.221906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221621    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:36.221906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221737    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:23:36.221906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221755    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.221906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.221774    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.222197 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.222107    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:36.223606 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.223573    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:23:36.223606 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.223576    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:23:36.223767 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.223622    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-kube-api-access-r7dqr" (OuterVolumeSpecName: "kube-api-access-r7dqr") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "kube-api-access-r7dqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:23:36.223767 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.223649    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb2d08dd-8d94-4d17-90f5-ab156c642b1e" (UID: "bb2d08dd-8d94-4d17-90f5-ab156c642b1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:23:36.323033 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.322999    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.323033 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.323028    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.323033 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.323038    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dqr\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-kube-api-access-r7dqr\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.323268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.323049    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.323268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.323058    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.323268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.323067    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.323268 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.323076    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08dd-8d94-4d17-90f5-ab156c642b1e-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:23:36.524949 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.524845    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"]
Aug 01 01:23:36.525250 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.525236    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy"
Aug 01 01:23:36.525295 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.525252    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy"
Aug 01 01:23:36.525295 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.525269    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy"
Aug 01 01:23:36.525295 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.525274    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy"
Aug 01 01:23:36.525394 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.525325    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy"
Aug 01 01:23:36.525394 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.525334    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerName="istio-proxy"
Aug 01 01:23:36.527385 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.527361    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:36.529735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.529709    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-423d15\""
Aug 01 01:23:36.529735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.529730    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-423d15-dockercfg-27c48\""
Aug 01 01:23:36.538722 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.538699    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"]
Aug 01 01:23:36.612566 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.612536    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"]
Aug 01 01:23:36.614895 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.614877    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:36.625140 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.625118    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"]
Aug 01 01:23:36.625482 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.625455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmhp\" (UniqueName: \"kubernetes.io/projected/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-kube-api-access-4nmhp\") pod \"payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5\" (UID: \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:36.625578 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.625526    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5\" (UID: \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:36.644371 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.644337    2570 generic.go:356] "Generic (PLEG): container finished" podID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" containerID="ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338" exitCode=0
Aug 01 01:23:36.644543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.644371    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" event={"ID":"bb2d08dd-8d94-4d17-90f5-ab156c642b1e","Type":"ContainerDied","Data":"ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338"}
Aug 01 01:23:36.644543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.644440    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672" event={"ID":"bb2d08dd-8d94-4d17-90f5-ab156c642b1e","Type":"ContainerDied","Data":"be2df7776ee65b225576c29d263e584957c09f99960c0fa748fba8b3e03025d1"}
Aug 01 01:23:36.644543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.644466    2570 scope.go:117] "RemoveContainer" containerID="ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338"
Aug 01 01:23:36.644750 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.644656    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"
Aug 01 01:23:36.655528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.655510    2570 scope.go:117] "RemoveContainer" containerID="5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69"
Aug 01 01:23:36.661583 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.661562    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"]
Aug 01 01:23:36.663197 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.663174    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a9f952-openshift-default-66d77dd4bb-2r672"]
Aug 01 01:23:36.664301 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.664286    2570 scope.go:117] "RemoveContainer" containerID="ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338"
Aug 01 01:23:36.664572 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:23:36.664553    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338\": container with ID starting with ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338 not found: ID does not exist" containerID="ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338"
Aug 01 01:23:36.664655 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.664585    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338"} err="failed to get container status \"ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338\": rpc error: code = NotFound desc = could not find container \"ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338\": container with ID starting with ab3d3fdec8075e4a83dd012daff709a30af07e9a6b8e507b55a2125d94ad5338 not found: ID does not exist"
Aug 01 01:23:36.664655 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.664611    2570 scope.go:117] "RemoveContainer" containerID="5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69"
Aug 01 01:23:36.664902 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:23:36.664885    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69\": container with ID starting with 5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69 not found: ID does not exist" containerID="5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69"
Aug 01 01:23:36.664961 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.664908    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69"} err="failed to get container status \"5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69\": rpc error: code = NotFound desc = could not find container \"5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69\": container with ID starting with 5d7f9fbed519553bdf5c89323daa18471f659e2d251735599c6ef976be4b3f69 not found: ID does not exist"
Aug 01 01:23:36.726062 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.726034    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/147bcadf-ee13-4c01-a561-0946523c5445-kube-api-access-g9j8n\") pod \"payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f\" (UID: \"147bcadf-ee13-4c01-a561-0946523c5445\") " pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:36.726166 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.726122    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmhp\" (UniqueName: \"kubernetes.io/projected/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-kube-api-access-4nmhp\") pod \"payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5\" (UID: \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:36.726166 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.726162    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5\" (UID: \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:36.726260 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.726213    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147bcadf-ee13-4c01-a561-0946523c5445-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f\" (UID: \"147bcadf-ee13-4c01-a561-0946523c5445\") " pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:36.726782 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.726766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5\" (UID: \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:36.734149 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.734125    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmhp\" (UniqueName: \"kubernetes.io/projected/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-kube-api-access-4nmhp\") pod \"payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5\" (UID: \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:36.827324 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.827295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147bcadf-ee13-4c01-a561-0946523c5445-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f\" (UID: \"147bcadf-ee13-4c01-a561-0946523c5445\") " pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:36.827470 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.827336    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/147bcadf-ee13-4c01-a561-0946523c5445-kube-api-access-g9j8n\") pod \"payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f\" (UID: \"147bcadf-ee13-4c01-a561-0946523c5445\") " pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:36.828062 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.828041    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147bcadf-ee13-4c01-a561-0946523c5445-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f\" (UID: \"147bcadf-ee13-4c01-a561-0946523c5445\") " pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:36.835264 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.835238    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/147bcadf-ee13-4c01-a561-0946523c5445-kube-api-access-g9j8n\") pod \"payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f\" (UID: \"147bcadf-ee13-4c01-a561-0946523c5445\") " pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:36.838024 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.838008    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:36.926721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:36.926686    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:37.059655 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.059630    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"]
Aug 01 01:23:37.061830 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:23:37.061804    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147bcadf_ee13_4c01_a561_0946523c5445.slice/crio-4278c9d87a3119b0cd0600214dc526ee698b452c27207e607924328d6c815c6e WatchSource:0}: Error finding container 4278c9d87a3119b0cd0600214dc526ee698b452c27207e607924328d6c815c6e: Status 404 returned error can't find the container with id 4278c9d87a3119b0cd0600214dc526ee698b452c27207e607924328d6c815c6e
Aug 01 01:23:37.177245 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.177214    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"]
Aug 01 01:23:37.181644 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:23:37.181591    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8154cdf2_3ec9_4602_bdb9_fe7e60b5a570.slice/crio-ef9dab88b84515f31888885de21bcf1b685928e1445aff4196d06a9cb302ab47 WatchSource:0}: Error finding container ef9dab88b84515f31888885de21bcf1b685928e1445aff4196d06a9cb302ab47: Status 404 returned error can't find the container with id ef9dab88b84515f31888885de21bcf1b685928e1445aff4196d06a9cb302ab47
Aug 01 01:23:37.652432 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.652389    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f" event={"ID":"147bcadf-ee13-4c01-a561-0946523c5445","Type":"ContainerStarted","Data":"51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd"}
Aug 01 01:23:37.652631 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.652453    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f" event={"ID":"147bcadf-ee13-4c01-a561-0946523c5445","Type":"ContainerStarted","Data":"4278c9d87a3119b0cd0600214dc526ee698b452c27207e607924328d6c815c6e"}
Aug 01 01:23:37.652631 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.652589    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:37.653997 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.653965    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5" event={"ID":"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570","Type":"ContainerStarted","Data":"377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399"}
Aug 01 01:23:37.654131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.654009    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5" event={"ID":"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570","Type":"ContainerStarted","Data":"ef9dab88b84515f31888885de21bcf1b685928e1445aff4196d06a9cb302ab47"}
Aug 01 01:23:37.654131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.654112    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:37.675561 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.675511    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f" podStartSLOduration=1.675495 podStartE2EDuration="1.675495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:23:37.672016437 +0000 UTC m=+2983.905948706" watchObservedRunningTime="2026-08-01 01:23:37.675495 +0000 UTC m=+2983.909427267"
Aug 01 01:23:37.687533 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:37.687493    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5" podStartSLOduration=1.687479811 podStartE2EDuration="1.687479811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:23:37.68570732 +0000 UTC m=+2983.919639584" watchObservedRunningTime="2026-08-01 01:23:37.687479811 +0000 UTC m=+2983.921412078"
Aug 01 01:23:38.262375 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:38.262340    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2d08dd-8d94-4d17-90f5-ab156c642b1e" path="/var/lib/kubelet/pods/bb2d08dd-8d94-4d17-90f5-ab156c642b1e/volumes"
Aug 01 01:23:48.660821 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:48.660787    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:23:48.661324 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:48.660993    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:23:48.928396 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:48.928356    2570 patch_prober.go:28] interesting pod/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:41134->10.133.0.82:15021: read: connection reset by peer" start-of-body=
Aug 01 01:23:48.928564 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:48.928436    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:41134->10.133.0.82:15021: read: connection reset by peer"
Aug 01 01:23:49.719332 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:49.719297    2570 generic.go:356] "Generic (PLEG): container finished" podID="ab752358-b918-4050-98df-3c17df616610" containerID="303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299" exitCode=0
Aug 01 01:23:49.719714 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:49.719375    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" event={"ID":"ab752358-b918-4050-98df-3c17df616610","Type":"ContainerDied","Data":"303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299"}
Aug 01 01:23:49.719760 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:49.719746    2570 scope.go:117] "RemoveContainer" containerID="303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299"
Aug 01 01:23:49.720675 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:49.720644    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:23:49.720722 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:49.720713    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:23:49.720773 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:49.720762    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:23:50.726409 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:50.726374    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" event={"ID":"ab752358-b918-4050-98df-3c17df616610","Type":"ContainerStarted","Data":"283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3"}
Aug 01 01:23:51.161199 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:51.161160    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:51.161430 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:51.161215    2570 patch_prober.go:28] interesting pod/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Aug 01 01:23:51.161430 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:51.161283    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Aug 01 01:23:52.161432 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:52.161387    2570 patch_prober.go:28] interesting pod/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Aug 01 01:23:52.162001 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:52.161456    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Aug 01 01:23:53.161508 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:53.161464    2570 patch_prober.go:28] interesting pod/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Aug 01 01:23:53.162142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:53.162110    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Aug 01 01:23:54.165419 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:54.165382    2570 patch_prober.go:28] interesting pod/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Aug 01 01:23:54.165892 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:54.165447    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Aug 01 01:23:55.161150 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:55.161112    2570 patch_prober.go:28] interesting pod/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Aug 01 01:23:55.161467 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:55.161430    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Aug 01 01:23:56.163809 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:56.163778    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:56.164222 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:56.164118    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:23:56.164834 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:23:56.164817    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:24:00.341726 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.341693    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"]
Aug 01 01:24:00.345340 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.345311    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.347858 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.347832    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-65669d-openshift-default-dockercfg-cd7nt\""
Aug 01 01:24:00.356058 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.356031    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"]
Aug 01 01:24:00.439738 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.439705    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-data\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.439738 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.439739    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-socket\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.440030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.439765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-credential-socket\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.440030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.439834    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01c24-0eac-420c-abd0-bbf9912def63-istio-podinfo\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.440030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.439878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01c24-0eac-420c-abd0-bbf9912def63-istiod-ca-cert\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.440030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.439919    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-istio-token\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.440030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.439980    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9bz\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-kube-api-access-6x9bz\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.440030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.440018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-envoy\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.440336 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.440035    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-certs\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.540536 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.540499    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01c24-0eac-420c-abd0-bbf9912def63-istio-podinfo\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.540536 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.540534    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01c24-0eac-420c-abd0-bbf9912def63-istiod-ca-cert\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.540810 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.540565    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-istio-token\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.540810 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.540694    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9bz\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-kube-api-access-6x9bz\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.540810 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.540768    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-envoy\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.540810 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.540798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-certs\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.541062 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.540903    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-data\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.541062 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.541040    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-socket\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.541171 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.541116    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-credential-socket\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.541235 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.541202    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-certs\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.541387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.541364    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01c24-0eac-420c-abd0-bbf9912def63-istiod-ca-cert\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.541459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.541401    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-socket\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.541459 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.541446    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-credential-socket\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.541529 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.541505    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-data\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.543104 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.543076    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-envoy\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.543222 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.543161    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01c24-0eac-420c-abd0-bbf9912def63-istio-podinfo\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.548034 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.548010    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9bz\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-kube-api-access-6x9bz\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.548139 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.548014    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-istio-token\") pod \"e2e-shared-b-65669d-openshift-default-8f4884544-jczpf\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") " pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.657772 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.657691    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:00.789259 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.789231    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"]
Aug 01 01:24:00.791132 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:24:00.791101    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c01c24_0eac_420c_abd0_bbf9912def63.slice/crio-bdd32497ef4dea8949be3830611f3912fd09e087214b6b1ddb9aa2e79dc83522 WatchSource:0}: Error finding container bdd32497ef4dea8949be3830611f3912fd09e087214b6b1ddb9aa2e79dc83522: Status 404 returned error can't find the container with id bdd32497ef4dea8949be3830611f3912fd09e087214b6b1ddb9aa2e79dc83522
Aug 01 01:24:00.793370 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.793331    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:24:00.793439 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.793420    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:24:00.793483 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:00.793468    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:24:01.779620 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:01.779577    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" event={"ID":"45c01c24-0eac-420c-abd0-bbf9912def63","Type":"ContainerStarted","Data":"6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09"}
Aug 01 01:24:01.780042 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:01.779639    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" event={"ID":"45c01c24-0eac-420c-abd0-bbf9912def63","Type":"ContainerStarted","Data":"bdd32497ef4dea8949be3830611f3912fd09e087214b6b1ddb9aa2e79dc83522"}
Aug 01 01:24:01.799085 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:01.799038    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" podStartSLOduration=1.7990219280000002 podStartE2EDuration="1.799021928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:24:01.797175794 +0000 UTC m=+3008.031108060" watchObservedRunningTime="2026-08-01 01:24:01.799021928 +0000 UTC m=+3008.032954194"
Aug 01 01:24:02.658080 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:02.658037    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:02.663051 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:02.663027    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:02.784161 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:02.784130    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:02.785032 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:02.785012    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:04.827142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:04.827110    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"]
Aug 01 01:24:04.830267 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:04.830243    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:04.832647 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:04.832618    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-65669d-serving-cert\""
Aug 01 01:24:04.841380 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:04.841355    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"]
Aug 01 01:24:04.978828 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:04.978795    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjlq\" (UniqueName: \"kubernetes.io/projected/9474041b-b9fb-4647-a03f-4b448599a5b8-kube-api-access-vvjlq\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:04.979021 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:04.978839    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9474041b-b9fb-4647-a03f-4b448599a5b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:04.979021 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:04.978865    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.079648 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.079572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjlq\" (UniqueName: \"kubernetes.io/projected/9474041b-b9fb-4647-a03f-4b448599a5b8-kube-api-access-vvjlq\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.079648 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.079615    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9474041b-b9fb-4647-a03f-4b448599a5b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.079648 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.079642    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.079876 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:24:05.079805    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-serving-cert: secret "maas-api-e2e-shared-b-65669d-serving-cert" not found
Aug 01 01:24:05.079916 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:24:05.079878    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls podName:9474041b-b9fb-4647-a03f-4b448599a5b8 nodeName:}" failed. No retries permitted until 2026-08-01 01:24:05.57986205 +0000 UTC m=+3011.813794295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls") pod "maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj" (UID: "9474041b-b9fb-4647-a03f-4b448599a5b8") : secret "maas-api-e2e-shared-b-65669d-serving-cert" not found
Aug 01 01:24:05.080305 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.080286    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9474041b-b9fb-4647-a03f-4b448599a5b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.089721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.089700    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjlq\" (UniqueName: \"kubernetes.io/projected/9474041b-b9fb-4647-a03f-4b448599a5b8-kube-api-access-vvjlq\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.466036 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.465955    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"]
Aug 01 01:24:05.469327 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.469312    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:05.471700 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.471682    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-65669d-dockercfg-8c265\""
Aug 01 01:24:05.471794 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.471719    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-65669d\""
Aug 01 01:24:05.478987 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.478961    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"]
Aug 01 01:24:05.584043 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.584008    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtl5r\" (UniqueName: \"kubernetes.io/projected/eb36149e-d787-4487-9408-33265e84bdf4-kube-api-access-wtl5r\") pod \"payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk\" (UID: \"eb36149e-d787-4487-9408-33265e84bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:05.584244 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.584070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.584244 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.584106    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36149e-d787-4487-9408-33265e84bdf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk\" (UID: \"eb36149e-d787-4487-9408-33265e84bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:05.586831 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.586799    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.685097 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.685063    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36149e-d787-4487-9408-33265e84bdf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk\" (UID: \"eb36149e-d787-4487-9408-33265e84bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:05.685257 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.685159    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtl5r\" (UniqueName: \"kubernetes.io/projected/eb36149e-d787-4487-9408-33265e84bdf4-kube-api-access-wtl5r\") pod \"payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk\" (UID: \"eb36149e-d787-4487-9408-33265e84bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:05.685764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.685740    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36149e-d787-4487-9408-33265e84bdf4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk\" (UID: \"eb36149e-d787-4487-9408-33265e84bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:05.692669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.692642    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtl5r\" (UniqueName: \"kubernetes.io/projected/eb36149e-d787-4487-9408-33265e84bdf4-kube-api-access-wtl5r\") pod \"payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk\" (UID: \"eb36149e-d787-4487-9408-33265e84bdf4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:05.742543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.742470    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:05.781654 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.781616    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:05.890829 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.890790    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"]
Aug 01 01:24:05.892730 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:24:05.892703    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9474041b_b9fb_4647_a03f_4b448599a5b8.slice/crio-3b4472c16f4a403377b91858edbd41bd9abe4bceafe156bff7e2bc3b8ee59d91 WatchSource:0}: Error finding container 3b4472c16f4a403377b91858edbd41bd9abe4bceafe156bff7e2bc3b8ee59d91: Status 404 returned error can't find the container with id 3b4472c16f4a403377b91858edbd41bd9abe4bceafe156bff7e2bc3b8ee59d91
Aug 01 01:24:05.931985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:05.931958    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"]
Aug 01 01:24:05.933756 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:24:05.933732    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb36149e_d787_4487_9408_33265e84bdf4.slice/crio-294653c050a7abc8c49d58f8a6c3851a13b39ed044262f95babdbb657b291207 WatchSource:0}: Error finding container 294653c050a7abc8c49d58f8a6c3851a13b39ed044262f95babdbb657b291207: Status 404 returned error can't find the container with id 294653c050a7abc8c49d58f8a6c3851a13b39ed044262f95babdbb657b291207
Aug 01 01:24:06.805312 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:06.805268    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk" event={"ID":"eb36149e-d787-4487-9408-33265e84bdf4","Type":"ContainerStarted","Data":"b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757"}
Aug 01 01:24:06.805510 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:06.805331    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk" event={"ID":"eb36149e-d787-4487-9408-33265e84bdf4","Type":"ContainerStarted","Data":"294653c050a7abc8c49d58f8a6c3851a13b39ed044262f95babdbb657b291207"}
Aug 01 01:24:06.805510 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:06.805451    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:06.806762 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:06.806736    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj" event={"ID":"9474041b-b9fb-4647-a03f-4b448599a5b8","Type":"ContainerStarted","Data":"4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c"}
Aug 01 01:24:06.806884 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:06.806786    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj" event={"ID":"9474041b-b9fb-4647-a03f-4b448599a5b8","Type":"ContainerStarted","Data":"3b4472c16f4a403377b91858edbd41bd9abe4bceafe156bff7e2bc3b8ee59d91"}
Aug 01 01:24:06.806884 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:06.806815    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:06.820714 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:06.820673    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk" podStartSLOduration=1.820660815 podStartE2EDuration="1.820660815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:24:06.819843265 +0000 UTC m=+3013.053775531" watchObservedRunningTime="2026-08-01 01:24:06.820660815 +0000 UTC m=+3013.054593080"
Aug 01 01:24:06.835615 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:06.835575    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj" podStartSLOduration=2.421331284 podStartE2EDuration="2.835563339s" totalImagesPullingTime="414.232055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:24:06.834716177 +0000 UTC m=+3013.068648437" watchObservedRunningTime="2026-08-01 01:24:06.835563339 +0000 UTC m=+3013.069495605"
Aug 01 01:24:12.818266 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:12.818232    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:24:17.786122 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.786034    2570 patch_prober.go:28] interesting pod/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Aug 01 01:24:17.786122 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.786097    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Aug 01 01:24:17.814360 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.814335    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:24:17.863739 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.863708    2570 generic.go:356] "Generic (PLEG): container finished" podID="45c01c24-0eac-420c-abd0-bbf9912def63" containerID="6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09" exitCode=0
Aug 01 01:24:17.863905 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.863743    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" event={"ID":"45c01c24-0eac-420c-abd0-bbf9912def63","Type":"ContainerDied","Data":"6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09"}
Aug 01 01:24:17.864089 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.864071    2570 scope.go:117] "RemoveContainer" containerID="6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09"
Aug 01 01:24:17.865082 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.865054    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:24:17.865161 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.865121    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:24:17.865161 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:17.865149    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:24:18.874167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:18.874119    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" event={"ID":"45c01c24-0eac-420c-abd0-bbf9912def63","Type":"ContainerStarted","Data":"b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7"}
Aug 01 01:24:19.658955 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:19.658455    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:19.658955 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:19.658586    2570 patch_prober.go:28] interesting pod/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Aug 01 01:24:19.658955 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:19.658634    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Aug 01 01:24:20.658351 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:20.658311    2570 patch_prober.go:28] interesting pod/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Aug 01 01:24:20.658952 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:20.658900    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Aug 01 01:24:21.658343 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:21.658303    2570 patch_prober.go:28] interesting pod/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Aug 01 01:24:21.658633 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:21.658604    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Aug 01 01:24:22.658818 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:22.658372    2570 patch_prober.go:28] interesting pod/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Aug 01 01:24:22.658818 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:22.658438    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Aug 01 01:24:23.658378 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:23.658337    2570 patch_prober.go:28] interesting pod/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Aug 01 01:24:23.658678 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:23.658646    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Aug 01 01:24:24.661879 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:24.661851    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:24.662275 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:24.662213    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:24.662897 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:24.662879    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:24:52.102640 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.102551    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"]
Aug 01 01:24:52.105550 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.105525    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.108304 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.108281    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"kube-root-ca.crt\""
Aug 01 01:24:52.108434 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.108398    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"openshift-service-ca.crt\""
Aug 01 01:24:52.109387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.109370    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"default-dockercfg-pslct\""
Aug 01 01:24:52.109477 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.109424    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"auth-test-model-423d15-kserve-self-signed-certs\""
Aug 01 01:24:52.123069 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.123042    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"]
Aug 01 01:24:52.189392 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.189359    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-home\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.189392 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.189393    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tmp-dir\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.189591 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.189430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tls-certs\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.189591 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.189540    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-dshm\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.189591 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.189578    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbl9\" (UniqueName: \"kubernetes.io/projected/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-kube-api-access-zqbl9\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.189736 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.189603    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-model-cache\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.290976 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.290925    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-dshm\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.290976 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.290978    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbl9\" (UniqueName: \"kubernetes.io/projected/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-kube-api-access-zqbl9\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.291207 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.291076    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-model-cache\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.291273 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.291228    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-home\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.291330 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.291270    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tmp-dir\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.291386 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.291349    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tls-certs\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.291456 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.291436    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-model-cache\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.291561 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.291545    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-home\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.291680 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.291653    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tmp-dir\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.293294 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.293273    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-dshm\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.293692 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.293674    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tls-certs\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.299976 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.299954    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbl9\" (UniqueName: \"kubernetes.io/projected/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-kube-api-access-zqbl9\") pod \"auth-test-model-423d15-kserve-796bb74b79-lqrxg\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") " pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.418377 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.418298    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:24:52.550369 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:52.550344    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"]
Aug 01 01:24:52.552661 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:24:52.552627    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cfd707c_c26b_4cf8_a04d_ebc847d602f7.slice/crio-a21dbd7070d27ecd5bed6a08b1060e2006d944b26197064920cd57b72fdb4545 WatchSource:0}: Error finding container a21dbd7070d27ecd5bed6a08b1060e2006d944b26197064920cd57b72fdb4545: Status 404 returned error can't find the container with id a21dbd7070d27ecd5bed6a08b1060e2006d944b26197064920cd57b72fdb4545
Aug 01 01:24:53.038083 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:53.038046    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg" event={"ID":"6cfd707c-c26b-4cf8-a04d-ebc847d602f7","Type":"ContainerStarted","Data":"670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3"}
Aug 01 01:24:53.038249 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:53.038092    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg" event={"ID":"6cfd707c-c26b-4cf8-a04d-ebc847d602f7","Type":"ContainerStarted","Data":"a21dbd7070d27ecd5bed6a08b1060e2006d944b26197064920cd57b72fdb4545"}
Aug 01 01:24:53.054159 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:24:53.054106    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg" podStartSLOduration=1.0540899989999999 podStartE2EDuration="1.054089999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:24:53.053863738 +0000 UTC m=+3059.287796004" watchObservedRunningTime="2026-08-01 01:24:53.054089999 +0000 UTC m=+3059.288022283"
Aug 01 01:25:02.418883 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:02.418844    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:25:02.418883 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:02.418898    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:25:02.432032 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:02.432002    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:25:03.097547 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:03.097514    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:25:16.297264 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.297229    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"]
Aug 01 01:25:16.299665 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.299648    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.302505 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.302486    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"openshift-service-ca.crt\""
Aug 01 01:25:16.303413 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.303396    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"auth-test-model-65669d-kserve-self-signed-certs\""
Aug 01 01:25:16.303495 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.303432    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"kube-root-ca.crt\""
Aug 01 01:25:16.303574 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.303553    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"default-dockercfg-f4wd6\""
Aug 01 01:25:16.308751 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.308727    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"]
Aug 01 01:25:16.399128 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.399105    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-home\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.399254 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.399133    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvbj\" (UniqueName: \"kubernetes.io/projected/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-kube-api-access-zmvbj\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.399254 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.399162    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tls-certs\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.399254 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.399243    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tmp-dir\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.399430 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.399346    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-dshm\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.399430 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.399385    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-model-cache\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.500506 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.500475    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tmp-dir\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.500643 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.500541    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-dshm\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.500643 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.500570    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-model-cache\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.500643 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.500628    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-home\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.500815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.500658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvbj\" (UniqueName: \"kubernetes.io/projected/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-kube-api-access-zmvbj\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.500815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.500709    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tls-certs\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.500924 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.500900    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tmp-dir\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.501024 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.500977    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-model-cache\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.501143 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.501054    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-home\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.502851 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.502831    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-dshm\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.503196 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.503178    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tls-certs\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.508325 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.508306    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvbj\" (UniqueName: \"kubernetes.io/projected/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-kube-api-access-zmvbj\") pod \"auth-test-model-65669d-kserve-647d66f855-46sq5\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") " pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.609792 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.609761    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:16.742883 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:16.742856    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"]
Aug 01 01:25:16.744908 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:25:16.744869    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5fbc4d6_2562_40b5_abbd_a5f5567bdd16.slice/crio-ab8273484f7297bacca9b02334281717d06fd204f35c8256cd4cb72455505376 WatchSource:0}: Error finding container ab8273484f7297bacca9b02334281717d06fd204f35c8256cd4cb72455505376: Status 404 returned error can't find the container with id ab8273484f7297bacca9b02334281717d06fd204f35c8256cd4cb72455505376
Aug 01 01:25:17.156744 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:17.156704    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5" event={"ID":"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16","Type":"ContainerStarted","Data":"f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661"}
Aug 01 01:25:17.156894 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:17.156753    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5" event={"ID":"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16","Type":"ContainerStarted","Data":"ab8273484f7297bacca9b02334281717d06fd204f35c8256cd4cb72455505376"}
Aug 01 01:25:17.175942 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:17.175880    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5" podStartSLOduration=1.175865958 podStartE2EDuration="1.175865958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:25:17.173706452 +0000 UTC m=+3083.407638720" watchObservedRunningTime="2026-08-01 01:25:17.175865958 +0000 UTC m=+3083.409798224"
Aug 01 01:25:26.610545 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:26.610508    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:26.611150 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:26.610563    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:26.623036 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:26.623015    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:25:27.216352 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:25:27.216320    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:28:28.713381 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:28.713346    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"]
Aug 01 01:28:29.805632 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:29.805597    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"]
Aug 01 01:28:43.732635 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:43.732566    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg" podUID="6cfd707c-c26b-4cf8-a04d-ebc847d602f7" containerName="main" containerID="cri-o://670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3" gracePeriod=45
Aug 01 01:28:43.980365 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:43.980342    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:28:44.034618 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.034533    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-home\") pod \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") "
Aug 01 01:28:44.034618 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.034591    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqbl9\" (UniqueName: \"kubernetes.io/projected/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-kube-api-access-zqbl9\") pod \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") "
Aug 01 01:28:44.034815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.034679    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tmp-dir\") pod \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") "
Aug 01 01:28:44.034815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.034699    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tls-certs\") pod \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") "
Aug 01 01:28:44.034815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.034716    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-dshm\") pod \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") "
Aug 01 01:28:44.034815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.034744    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-model-cache\") pod \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\" (UID: \"6cfd707c-c26b-4cf8-a04d-ebc847d602f7\") "
Aug 01 01:28:44.035078 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.034831    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-home" (OuterVolumeSpecName: "home") pod "6cfd707c-c26b-4cf8-a04d-ebc847d602f7" (UID: "6cfd707c-c26b-4cf8-a04d-ebc847d602f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:28:44.035078 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.034979    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cfd707c-c26b-4cf8-a04d-ebc847d602f7" (UID: "6cfd707c-c26b-4cf8-a04d-ebc847d602f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:28:44.035078 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.035070    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:44.035221 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.035078    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cfd707c-c26b-4cf8-a04d-ebc847d602f7" (UID: "6cfd707c-c26b-4cf8-a04d-ebc847d602f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:28:44.035221 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.035088    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:44.036999 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.036972    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-dshm" (OuterVolumeSpecName: "dshm") pod "6cfd707c-c26b-4cf8-a04d-ebc847d602f7" (UID: "6cfd707c-c26b-4cf8-a04d-ebc847d602f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:28:44.037096 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.037035    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-kube-api-access-zqbl9" (OuterVolumeSpecName: "kube-api-access-zqbl9") pod "6cfd707c-c26b-4cf8-a04d-ebc847d602f7" (UID: "6cfd707c-c26b-4cf8-a04d-ebc847d602f7"). InnerVolumeSpecName "kube-api-access-zqbl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:28:44.037142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.037092    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cfd707c-c26b-4cf8-a04d-ebc847d602f7" (UID: "6cfd707c-c26b-4cf8-a04d-ebc847d602f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:28:44.136392 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.136354    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:44.136392 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.136383    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:44.136392 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.136397    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:44.136626 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.136409    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqbl9\" (UniqueName: \"kubernetes.io/projected/6cfd707c-c26b-4cf8-a04d-ebc847d602f7-kube-api-access-zqbl9\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:44.184481 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.184448    2570 generic.go:356] "Generic (PLEG): container finished" podID="6cfd707c-c26b-4cf8-a04d-ebc847d602f7" containerID="670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3" exitCode=0
Aug 01 01:28:44.184669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.184502    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"
Aug 01 01:28:44.184669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.184544    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg" event={"ID":"6cfd707c-c26b-4cf8-a04d-ebc847d602f7","Type":"ContainerDied","Data":"670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3"}
Aug 01 01:28:44.184669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.184592    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg" event={"ID":"6cfd707c-c26b-4cf8-a04d-ebc847d602f7","Type":"ContainerDied","Data":"a21dbd7070d27ecd5bed6a08b1060e2006d944b26197064920cd57b72fdb4545"}
Aug 01 01:28:44.184669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.184614    2570 scope.go:117] "RemoveContainer" containerID="670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3"
Aug 01 01:28:44.194081 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.194062    2570 scope.go:117] "RemoveContainer" containerID="670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3"
Aug 01 01:28:44.194346 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:28:44.194326    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3\": container with ID starting with 670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3 not found: ID does not exist" containerID="670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3"
Aug 01 01:28:44.194409 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.194353    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3"} err="failed to get container status \"670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3\": rpc error: code = NotFound desc = could not find container \"670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3\": container with ID starting with 670b12cf2b2d504f9004a3944bc36107e1e43958e3ec3eeb540b4e197681b1c3 not found: ID does not exist"
Aug 01 01:28:44.207660 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.207635    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"]
Aug 01 01:28:44.211271 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.211248    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/auth-test-model-423d15-kserve-796bb74b79-lqrxg"]
Aug 01 01:28:44.262623 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.262592    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cfd707c-c26b-4cf8-a04d-ebc847d602f7" path="/var/lib/kubelet/pods/6cfd707c-c26b-4cf8-a04d-ebc847d602f7/volumes"
Aug 01 01:28:44.825308 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:44.825242    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5" podUID="d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" containerName="main" containerID="cri-o://f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661" gracePeriod=45
Aug 01 01:28:45.072753 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.072729    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:28:45.147943 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.147847    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmvbj\" (UniqueName: \"kubernetes.io/projected/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-kube-api-access-zmvbj\") pod \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") "
Aug 01 01:28:45.147943 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.147882    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tmp-dir\") pod \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") "
Aug 01 01:28:45.147943 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.147923    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tls-certs\") pod \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") "
Aug 01 01:28:45.148238 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.147964    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-home\") pod \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") "
Aug 01 01:28:45.148238 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.147984    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-model-cache\") pod \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") "
Aug 01 01:28:45.148238 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.148002    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-dshm\") pod \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\" (UID: \"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16\") "
Aug 01 01:28:45.148238 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.148193    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" (UID: "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:28:45.148411 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.148265    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-home" (OuterVolumeSpecName: "home") pod "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" (UID: "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:28:45.148411 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.148277    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" (UID: "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:28:45.148411 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.148321    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:45.150275 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.150241    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" (UID: "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:28:45.150405 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.150313    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-dshm" (OuterVolumeSpecName: "dshm") pod "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" (UID: "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:28:45.150405 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.150332    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-kube-api-access-zmvbj" (OuterVolumeSpecName: "kube-api-access-zmvbj") pod "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" (UID: "d5fbc4d6-2562-40b5-abbd-a5f5567bdd16"). InnerVolumeSpecName "kube-api-access-zmvbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:28:45.194738 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.194702    2570 generic.go:356] "Generic (PLEG): container finished" podID="d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" containerID="f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661" exitCode=0
Aug 01 01:28:45.194906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.194742    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"
Aug 01 01:28:45.194906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.194787    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5" event={"ID":"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16","Type":"ContainerDied","Data":"f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661"}
Aug 01 01:28:45.194906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.194839    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5" event={"ID":"d5fbc4d6-2562-40b5-abbd-a5f5567bdd16","Type":"ContainerDied","Data":"ab8273484f7297bacca9b02334281717d06fd204f35c8256cd4cb72455505376"}
Aug 01 01:28:45.194906 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.194864    2570 scope.go:117] "RemoveContainer" containerID="f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661"
Aug 01 01:28:45.204955 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.204911    2570 scope.go:117] "RemoveContainer" containerID="f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661"
Aug 01 01:28:45.205200 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:28:45.205181    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661\": container with ID starting with f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661 not found: ID does not exist" containerID="f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661"
Aug 01 01:28:45.205253 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.205211    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661"} err="failed to get container status \"f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661\": rpc error: code = NotFound desc = could not find container \"f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661\": container with ID starting with f5ee328772a75795983c2c0a7489e1828d07825ebfc8f3d098c812f5540d8661 not found: ID does not exist"
Aug 01 01:28:45.219391 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.219367    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"]
Aug 01 01:28:45.222743 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.222721    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/auth-test-model-65669d-kserve-647d66f855-46sq5"]
Aug 01 01:28:45.249363 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.249337    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvbj\" (UniqueName: \"kubernetes.io/projected/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-kube-api-access-zmvbj\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:45.249363 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.249362    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:45.249478 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.249373    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:45.249478 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.249383    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:45.249478 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:45.249394    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:28:46.262861 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:28:46.262821    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" path="/var/lib/kubelet/pods/d5fbc4d6-2562-40b5-abbd-a5f5567bdd16/volumes"
Aug 01 01:29:16.397450 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.397329    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"]
Aug 01 01:29:16.398103 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.397889    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cfd707c-c26b-4cf8-a04d-ebc847d602f7" containerName="main"
Aug 01 01:29:16.398103 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.397905    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cfd707c-c26b-4cf8-a04d-ebc847d602f7" containerName="main"
Aug 01 01:29:16.398103 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.397915    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" containerName="main"
Aug 01 01:29:16.398103 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.397923    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" containerName="main"
Aug 01 01:29:16.398103 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.398061    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fbc4d6-2562-40b5-abbd-a5f5567bdd16" containerName="main"
Aug 01 01:29:16.398103 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.398075    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cfd707c-c26b-4cf8-a04d-ebc847d602f7" containerName="main"
Aug 01 01:29:16.400907 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.400887    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.403385 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.403363    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"openshift-service-ca.crt\""
Aug 01 01:29:16.403549 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.403529    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"kube-root-ca.crt\""
Aug 01 01:29:16.404534 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.404513    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"sub-test-model-65669d-kserve-self-signed-certs\""
Aug 01 01:29:16.404641 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.404593    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"default-dockercfg-f4wd6\""
Aug 01 01:29:16.409233 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.409209    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"]
Aug 01 01:29:16.417795 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.417768    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-dshm\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.417899 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.417845    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-model-cache\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.417984 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.417914    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-tmp-dir\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.418050 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.417990    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-home\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.418050 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.418024    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtzq\" (UniqueName: \"kubernetes.io/projected/8409111c-5fb6-4ab2-9a88-e7661136cd35-kube-api-access-vjtzq\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.418161 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.418073    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8409111c-5fb6-4ab2-9a88-e7661136cd35-tls-certs\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.518951 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.518900    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-home\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.519144 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.518965    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtzq\" (UniqueName: \"kubernetes.io/projected/8409111c-5fb6-4ab2-9a88-e7661136cd35-kube-api-access-vjtzq\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.519144 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.519012    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8409111c-5fb6-4ab2-9a88-e7661136cd35-tls-certs\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.519144 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.519039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-dshm\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.519144 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.519083    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-model-cache\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.519144 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.519142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-tmp-dir\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.519430 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.519309    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-home\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.519502 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.519473    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-model-cache\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.519562 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.519521    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-tmp-dir\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.521309 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.521285    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-dshm\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.521595 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.521576    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8409111c-5fb6-4ab2-9a88-e7661136cd35-tls-certs\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.526842 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.526815    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtzq\" (UniqueName: \"kubernetes.io/projected/8409111c-5fb6-4ab2-9a88-e7661136cd35-kube-api-access-vjtzq\") pod \"sub-test-model-65669d-kserve-665d7946b7-pxdzf\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") " pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.713836 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.713741    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:16.841623 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:16.841596    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"]
Aug 01 01:29:16.844311 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:29:16.844284    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8409111c_5fb6_4ab2_9a88_e7661136cd35.slice/crio-32ea3b55ce89c7aec9d441109822077a7c530adbb2b2de4806f93f17cd813e44 WatchSource:0}: Error finding container 32ea3b55ce89c7aec9d441109822077a7c530adbb2b2de4806f93f17cd813e44: Status 404 returned error can't find the container with id 32ea3b55ce89c7aec9d441109822077a7c530adbb2b2de4806f93f17cd813e44
Aug 01 01:29:17.353394 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:17.353358    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf" event={"ID":"8409111c-5fb6-4ab2-9a88-e7661136cd35","Type":"ContainerStarted","Data":"de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824"}
Aug 01 01:29:17.353394 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:17.353401    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf" event={"ID":"8409111c-5fb6-4ab2-9a88-e7661136cd35","Type":"ContainerStarted","Data":"32ea3b55ce89c7aec9d441109822077a7c530adbb2b2de4806f93f17cd813e44"}
Aug 01 01:29:17.369528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:17.369483    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf" podStartSLOduration=1.369469894 podStartE2EDuration="1.369469894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:29:17.368467401 +0000 UTC m=+3323.602399669" watchObservedRunningTime="2026-08-01 01:29:17.369469894 +0000 UTC m=+3323.603402160"
Aug 01 01:29:26.714134 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:26.714094    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:26.716577 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:26.714154    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:26.726605 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:26.726582    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:29:27.410018 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:29:27.409978    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:30:27.785002 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:27.784960    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"]
Aug 01 01:30:42.803637 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:42.803598    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf" podUID="8409111c-5fb6-4ab2-9a88-e7661136cd35" containerName="main" containerID="cri-o://de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824" gracePeriod=45
Aug 01 01:30:43.045042 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.045017    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:30:43.189884 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.189846    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8409111c-5fb6-4ab2-9a88-e7661136cd35-tls-certs\") pod \"8409111c-5fb6-4ab2-9a88-e7661136cd35\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") "
Aug 01 01:30:43.190079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.189893    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-tmp-dir\") pod \"8409111c-5fb6-4ab2-9a88-e7661136cd35\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") "
Aug 01 01:30:43.190079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.189913    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjtzq\" (UniqueName: \"kubernetes.io/projected/8409111c-5fb6-4ab2-9a88-e7661136cd35-kube-api-access-vjtzq\") pod \"8409111c-5fb6-4ab2-9a88-e7661136cd35\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") "
Aug 01 01:30:43.190079 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.189954    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-dshm\") pod \"8409111c-5fb6-4ab2-9a88-e7661136cd35\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") "
Aug 01 01:30:43.190269 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.190077    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-model-cache\") pod \"8409111c-5fb6-4ab2-9a88-e7661136cd35\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") "
Aug 01 01:30:43.190269 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.190192    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-home\") pod \"8409111c-5fb6-4ab2-9a88-e7661136cd35\" (UID: \"8409111c-5fb6-4ab2-9a88-e7661136cd35\") "
Aug 01 01:30:43.190269 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.190193    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8409111c-5fb6-4ab2-9a88-e7661136cd35" (UID: "8409111c-5fb6-4ab2-9a88-e7661136cd35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:30:43.190414 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.190355    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-model-cache" (OuterVolumeSpecName: "model-cache") pod "8409111c-5fb6-4ab2-9a88-e7661136cd35" (UID: "8409111c-5fb6-4ab2-9a88-e7661136cd35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:30:43.190489 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.190466    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-home" (OuterVolumeSpecName: "home") pod "8409111c-5fb6-4ab2-9a88-e7661136cd35" (UID: "8409111c-5fb6-4ab2-9a88-e7661136cd35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:30:43.190590 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.190574    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:30:43.190632 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.190597    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:30:43.190632 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.190611    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:30:43.192101 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.192078    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8409111c-5fb6-4ab2-9a88-e7661136cd35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8409111c-5fb6-4ab2-9a88-e7661136cd35" (UID: "8409111c-5fb6-4ab2-9a88-e7661136cd35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:30:43.192186 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.192081    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-dshm" (OuterVolumeSpecName: "dshm") pod "8409111c-5fb6-4ab2-9a88-e7661136cd35" (UID: "8409111c-5fb6-4ab2-9a88-e7661136cd35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:30:43.192232 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.192181    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8409111c-5fb6-4ab2-9a88-e7661136cd35-kube-api-access-vjtzq" (OuterVolumeSpecName: "kube-api-access-vjtzq") pod "8409111c-5fb6-4ab2-9a88-e7661136cd35" (UID: "8409111c-5fb6-4ab2-9a88-e7661136cd35"). InnerVolumeSpecName "kube-api-access-vjtzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:30:43.291273 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.291245    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjtzq\" (UniqueName: \"kubernetes.io/projected/8409111c-5fb6-4ab2-9a88-e7661136cd35-kube-api-access-vjtzq\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:30:43.291273 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.291269    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8409111c-5fb6-4ab2-9a88-e7661136cd35-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:30:43.291435 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.291280    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8409111c-5fb6-4ab2-9a88-e7661136cd35-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:30:43.795172 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.795136    2570 generic.go:356] "Generic (PLEG): container finished" podID="8409111c-5fb6-4ab2-9a88-e7661136cd35" containerID="de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824" exitCode=0
Aug 01 01:30:43.795337 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.795174    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"
Aug 01 01:30:43.795337 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.795207    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf" event={"ID":"8409111c-5fb6-4ab2-9a88-e7661136cd35","Type":"ContainerDied","Data":"de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824"}
Aug 01 01:30:43.795337 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.795246    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf" event={"ID":"8409111c-5fb6-4ab2-9a88-e7661136cd35","Type":"ContainerDied","Data":"32ea3b55ce89c7aec9d441109822077a7c530adbb2b2de4806f93f17cd813e44"}
Aug 01 01:30:43.795337 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.795263    2570 scope.go:117] "RemoveContainer" containerID="de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824"
Aug 01 01:30:43.804165 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.804149    2570 scope.go:117] "RemoveContainer" containerID="de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824"
Aug 01 01:30:43.804444 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:30:43.804427    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824\": container with ID starting with de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824 not found: ID does not exist" containerID="de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824"
Aug 01 01:30:43.804486 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.804453    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824"} err="failed to get container status \"de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824\": rpc error: code = NotFound desc = could not find container \"de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824\": container with ID starting with de3a400dcd5c7b38da886d2e2b963ff848ee3b230a6f837cd6a0a60167dc3824 not found: ID does not exist"
Aug 01 01:30:43.817110 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.817087    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"]
Aug 01 01:30:43.822315 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:43.822296    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/sub-test-model-65669d-kserve-665d7946b7-pxdzf"]
Aug 01 01:30:44.262285 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:44.262256    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8409111c-5fb6-4ab2-9a88-e7661136cd35" path="/var/lib/kubelet/pods/8409111c-5fb6-4ab2-9a88-e7661136cd35/volumes"
Aug 01 01:30:48.300590 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.300505    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"]
Aug 01 01:30:48.301030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.300868    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8409111c-5fb6-4ab2-9a88-e7661136cd35" containerName="main"
Aug 01 01:30:48.301030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.300878    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8409111c-5fb6-4ab2-9a88-e7661136cd35" containerName="main"
Aug 01 01:30:48.301030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.300961    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8409111c-5fb6-4ab2-9a88-e7661136cd35" containerName="main"
Aug 01 01:30:48.305264 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.305246    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.308030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.308009    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"kube-root-ca.crt\""
Aug 01 01:30:48.309254 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.309199    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"rate-test-model-423d15-kserve-self-signed-certs\""
Aug 01 01:30:48.309368 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.309254    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"default-dockercfg-pslct\""
Aug 01 01:30:48.309542 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.309518    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"openshift-service-ca.crt\""
Aug 01 01:30:48.311701 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.311678    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"]
Aug 01 01:30:48.436100 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.436067    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bk2\" (UniqueName: \"kubernetes.io/projected/7f10c774-47cc-4e2d-9229-5d264514eaef-kube-api-access-z6bk2\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.436100 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.436103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f10c774-47cc-4e2d-9229-5d264514eaef-tls-certs\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.436300 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.436131    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-dshm\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.436300 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.436175    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-model-cache\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.436300 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.436268    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-home\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.436506 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.436487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-tmp-dir\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537357 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537324    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-tmp-dir\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537546 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537364    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bk2\" (UniqueName: \"kubernetes.io/projected/7f10c774-47cc-4e2d-9229-5d264514eaef-kube-api-access-z6bk2\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537546 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537384    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f10c774-47cc-4e2d-9229-5d264514eaef-tls-certs\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537546 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537408    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-dshm\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537546 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537514    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-model-cache\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537776 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537563    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-home\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537856 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537836    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-tmp-dir\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537907 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537883    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-model-cache\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.537965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.537923    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-home\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.539618 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.539592    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-dshm\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.539900 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.539883    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f10c774-47cc-4e2d-9229-5d264514eaef-tls-certs\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.544907 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.544887    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bk2\" (UniqueName: \"kubernetes.io/projected/7f10c774-47cc-4e2d-9229-5d264514eaef-kube-api-access-z6bk2\") pod \"rate-test-model-423d15-kserve-75d76c9c87-lg4c6\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") " pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.615942 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.615908    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:48.745318 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.745227    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"]
Aug 01 01:30:48.748034 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:30:48.748008    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f10c774_47cc_4e2d_9229_5d264514eaef.slice/crio-63c586f93e42c4693d1e653200d5f5e1872d62295cf56f2c091a5377ddc81eac WatchSource:0}: Error finding container 63c586f93e42c4693d1e653200d5f5e1872d62295cf56f2c091a5377ddc81eac: Status 404 returned error can't find the container with id 63c586f93e42c4693d1e653200d5f5e1872d62295cf56f2c091a5377ddc81eac
Aug 01 01:30:48.822409 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.822370    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6" event={"ID":"7f10c774-47cc-4e2d-9229-5d264514eaef","Type":"ContainerStarted","Data":"a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa"}
Aug 01 01:30:48.822551 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.822423    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6" event={"ID":"7f10c774-47cc-4e2d-9229-5d264514eaef","Type":"ContainerStarted","Data":"63c586f93e42c4693d1e653200d5f5e1872d62295cf56f2c091a5377ddc81eac"}
Aug 01 01:30:48.838142 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:48.838097    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6" podStartSLOduration=0.838081767 podStartE2EDuration="838.081767ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:30:48.837063338 +0000 UTC m=+3415.070995605" watchObservedRunningTime="2026-08-01 01:30:48.838081767 +0000 UTC m=+3415.072014034"
Aug 01 01:30:58.616029 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:58.615989    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:58.616462 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:58.616051    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:58.628610 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:58.628582    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:30:58.884556 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:30:58.884473    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:31:42.394028 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.393994    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"]
Aug 01 01:31:42.397333 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.397309    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.400589 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.400563    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"kube-root-ca.crt\""
Aug 01 01:31:42.400857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.400778    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"rate-test-model-65669d-kserve-self-signed-certs\""
Aug 01 01:31:42.400979 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.400963    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"default-dockercfg-f4wd6\""
Aug 01 01:31:42.401289 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.401262    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65669d\"/\"openshift-service-ca.crt\""
Aug 01 01:31:42.411476 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.411452    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"]
Aug 01 01:31:42.486882 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.486844    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tmp-dir\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.487086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.486900    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-model-cache\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.487086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.486948    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-dshm\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.487086 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.487052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tls-certs\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.487274 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.487099    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m74hj\" (UniqueName: \"kubernetes.io/projected/2f43ea69-ccc5-476d-9d09-b981ff0e1838-kube-api-access-m74hj\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.487274 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.487167    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-home\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.587790 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.587756    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tmp-dir\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.587988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.587803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-model-cache\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.587988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.587840    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-dshm\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.587988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.587861    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tls-certs\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.587988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.587885    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m74hj\" (UniqueName: \"kubernetes.io/projected/2f43ea69-ccc5-476d-9d09-b981ff0e1838-kube-api-access-m74hj\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.587988 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.587954    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-home\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.588231 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.588210    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tmp-dir\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.588297 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.588256    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-model-cache\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.588297 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.588284    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-home\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.590174 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.590152    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-dshm\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.590434 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.590418    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tls-certs\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.595584 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.595566    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m74hj\" (UniqueName: \"kubernetes.io/projected/2f43ea69-ccc5-476d-9d09-b981ff0e1838-kube-api-access-m74hj\") pod \"rate-test-model-65669d-kserve-784655f48c-9772d\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") " pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.713666 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.713583    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:42.845872 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:42.845846    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"]
Aug 01 01:31:42.848733 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:31:42.848713    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f43ea69_ccc5_476d_9d09_b981ff0e1838.slice/crio-267434be0cc6f7416710582de03dc7ce02329d36c5e26f3f6d359dcd1adc8ec7 WatchSource:0}: Error finding container 267434be0cc6f7416710582de03dc7ce02329d36c5e26f3f6d359dcd1adc8ec7: Status 404 returned error can't find the container with id 267434be0cc6f7416710582de03dc7ce02329d36c5e26f3f6d359dcd1adc8ec7
Aug 01 01:31:43.083553 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:43.083519    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d" event={"ID":"2f43ea69-ccc5-476d-9d09-b981ff0e1838","Type":"ContainerStarted","Data":"ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2"}
Aug 01 01:31:43.083734 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:43.083561    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d" event={"ID":"2f43ea69-ccc5-476d-9d09-b981ff0e1838","Type":"ContainerStarted","Data":"267434be0cc6f7416710582de03dc7ce02329d36c5e26f3f6d359dcd1adc8ec7"}
Aug 01 01:31:43.101890 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:43.101843    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d" podStartSLOduration=1.101827951 podStartE2EDuration="1.101827951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:31:43.099886149 +0000 UTC m=+3469.333818415" watchObservedRunningTime="2026-08-01 01:31:43.101827951 +0000 UTC m=+3469.335760248"
Aug 01 01:31:52.714683 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:52.714636    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:52.715178 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:52.714698    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:52.727357 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:52.727334    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:31:53.150128 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:31:53.150096    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:32:54.009277 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:32:54.009242    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"]
Aug 01 01:32:55.127989 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:32:55.127957    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"]
Aug 01 01:33:09.028832 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.028790    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6" podUID="7f10c774-47cc-4e2d-9229-5d264514eaef" containerName="main" containerID="cri-o://a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa" gracePeriod=45
Aug 01 01:33:09.281740 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.281683    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:33:09.463678 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.463642    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6bk2\" (UniqueName: \"kubernetes.io/projected/7f10c774-47cc-4e2d-9229-5d264514eaef-kube-api-access-z6bk2\") pod \"7f10c774-47cc-4e2d-9229-5d264514eaef\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") "
Aug 01 01:33:09.463875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.463700    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-tmp-dir\") pod \"7f10c774-47cc-4e2d-9229-5d264514eaef\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") "
Aug 01 01:33:09.463875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.463739    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-home\") pod \"7f10c774-47cc-4e2d-9229-5d264514eaef\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") "
Aug 01 01:33:09.463875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.463773    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-model-cache\") pod \"7f10c774-47cc-4e2d-9229-5d264514eaef\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") "
Aug 01 01:33:09.463875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.463821    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-dshm\") pod \"7f10c774-47cc-4e2d-9229-5d264514eaef\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") "
Aug 01 01:33:09.463875 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.463869    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f10c774-47cc-4e2d-9229-5d264514eaef-tls-certs\") pod \"7f10c774-47cc-4e2d-9229-5d264514eaef\" (UID: \"7f10c774-47cc-4e2d-9229-5d264514eaef\") "
Aug 01 01:33:09.464151 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.463996    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f10c774-47cc-4e2d-9229-5d264514eaef" (UID: "7f10c774-47cc-4e2d-9229-5d264514eaef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:33:09.464151 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.464018    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-home" (OuterVolumeSpecName: "home") pod "7f10c774-47cc-4e2d-9229-5d264514eaef" (UID: "7f10c774-47cc-4e2d-9229-5d264514eaef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:33:09.464151 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.464120    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f10c774-47cc-4e2d-9229-5d264514eaef" (UID: "7f10c774-47cc-4e2d-9229-5d264514eaef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:33:09.464306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.464233    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:09.464306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.464249    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:09.464306 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.464259    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:09.466014 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.465992    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f10c774-47cc-4e2d-9229-5d264514eaef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f10c774-47cc-4e2d-9229-5d264514eaef" (UID: "7f10c774-47cc-4e2d-9229-5d264514eaef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:33:09.466104 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.466044    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-dshm" (OuterVolumeSpecName: "dshm") pod "7f10c774-47cc-4e2d-9229-5d264514eaef" (UID: "7f10c774-47cc-4e2d-9229-5d264514eaef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:33:09.466157 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.466107    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f10c774-47cc-4e2d-9229-5d264514eaef-kube-api-access-z6bk2" (OuterVolumeSpecName: "kube-api-access-z6bk2") pod "7f10c774-47cc-4e2d-9229-5d264514eaef" (UID: "7f10c774-47cc-4e2d-9229-5d264514eaef"). InnerVolumeSpecName "kube-api-access-z6bk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:33:09.520576 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.520548    2570 generic.go:356] "Generic (PLEG): container finished" podID="7f10c774-47cc-4e2d-9229-5d264514eaef" containerID="a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa" exitCode=0
Aug 01 01:33:09.520684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.520589    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"
Aug 01 01:33:09.520684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.520608    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6" event={"ID":"7f10c774-47cc-4e2d-9229-5d264514eaef","Type":"ContainerDied","Data":"a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa"}
Aug 01 01:33:09.520684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.520640    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6" event={"ID":"7f10c774-47cc-4e2d-9229-5d264514eaef","Type":"ContainerDied","Data":"63c586f93e42c4693d1e653200d5f5e1872d62295cf56f2c091a5377ddc81eac"}
Aug 01 01:33:09.520684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.520658    2570 scope.go:117] "RemoveContainer" containerID="a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa"
Aug 01 01:33:09.529293 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.529274    2570 scope.go:117] "RemoveContainer" containerID="a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa"
Aug 01 01:33:09.529552 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:33:09.529533    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa\": container with ID starting with a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa not found: ID does not exist" containerID="a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa"
Aug 01 01:33:09.529591 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.529560    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa"} err="failed to get container status \"a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa\": rpc error: code = NotFound desc = could not find container \"a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa\": container with ID starting with a3afd233fcc0aee650fae274c42a3fb9f26371e77b9b06591b4af4f1e72819fa not found: ID does not exist"
Aug 01 01:33:09.540223 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.540178    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"]
Aug 01 01:33:09.543980 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.543943    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/rate-test-model-423d15-kserve-75d76c9c87-lg4c6"]
Aug 01 01:33:09.565341 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.565323    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bk2\" (UniqueName: \"kubernetes.io/projected/7f10c774-47cc-4e2d-9229-5d264514eaef-kube-api-access-z6bk2\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:09.565417 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.565343    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f10c774-47cc-4e2d-9229-5d264514eaef-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:09.565417 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:09.565355    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f10c774-47cc-4e2d-9229-5d264514eaef-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:10.146820 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.146780    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d" podUID="2f43ea69-ccc5-476d-9d09-b981ff0e1838" containerName="main" containerID="cri-o://ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2" gracePeriod=45
Aug 01 01:33:10.262721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.262697    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f10c774-47cc-4e2d-9229-5d264514eaef" path="/var/lib/kubelet/pods/7f10c774-47cc-4e2d-9229-5d264514eaef/volumes"
Aug 01 01:33:10.398492 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.398433    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:33:10.527332 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.527298    2570 generic.go:356] "Generic (PLEG): container finished" podID="2f43ea69-ccc5-476d-9d09-b981ff0e1838" containerID="ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2" exitCode=0
Aug 01 01:33:10.527514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.527344    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"
Aug 01 01:33:10.527514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.527387    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d" event={"ID":"2f43ea69-ccc5-476d-9d09-b981ff0e1838","Type":"ContainerDied","Data":"ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2"}
Aug 01 01:33:10.527514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.527437    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d" event={"ID":"2f43ea69-ccc5-476d-9d09-b981ff0e1838","Type":"ContainerDied","Data":"267434be0cc6f7416710582de03dc7ce02329d36c5e26f3f6d359dcd1adc8ec7"}
Aug 01 01:33:10.527514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.527456    2570 scope.go:117] "RemoveContainer" containerID="ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2"
Aug 01 01:33:10.535813 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.535797    2570 scope.go:117] "RemoveContainer" containerID="ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2"
Aug 01 01:33:10.536075 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:33:10.536055    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2\": container with ID starting with ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2 not found: ID does not exist" containerID="ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2"
Aug 01 01:33:10.536147 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.536086    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2"} err="failed to get container status \"ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2\": rpc error: code = NotFound desc = could not find container \"ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2\": container with ID starting with ee7049053fae9ae048660c209aaa94c2c81ec00ab2503bdfa5b41ec898a44be2 not found: ID does not exist"
Aug 01 01:33:10.572324 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572304    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tmp-dir\") pod \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") "
Aug 01 01:33:10.572412 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572351    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-model-cache\") pod \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") "
Aug 01 01:33:10.572412 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572372    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-dshm\") pod \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") "
Aug 01 01:33:10.572412 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572400    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-home\") pod \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") "
Aug 01 01:33:10.572554 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572419    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m74hj\" (UniqueName: \"kubernetes.io/projected/2f43ea69-ccc5-476d-9d09-b981ff0e1838-kube-api-access-m74hj\") pod \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") "
Aug 01 01:33:10.572554 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572457    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tls-certs\") pod \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\" (UID: \"2f43ea69-ccc5-476d-9d09-b981ff0e1838\") "
Aug 01 01:33:10.572657 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572561    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f43ea69-ccc5-476d-9d09-b981ff0e1838" (UID: "2f43ea69-ccc5-476d-9d09-b981ff0e1838"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:33:10.572657 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572583    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f43ea69-ccc5-476d-9d09-b981ff0e1838" (UID: "2f43ea69-ccc5-476d-9d09-b981ff0e1838"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:33:10.572758 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572694    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-home" (OuterVolumeSpecName: "home") pod "2f43ea69-ccc5-476d-9d09-b981ff0e1838" (UID: "2f43ea69-ccc5-476d-9d09-b981ff0e1838"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:33:10.572758 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572742    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:10.572851 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.572762    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:10.574520 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.574497    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f43ea69-ccc5-476d-9d09-b981ff0e1838-kube-api-access-m74hj" (OuterVolumeSpecName: "kube-api-access-m74hj") pod "2f43ea69-ccc5-476d-9d09-b981ff0e1838" (UID: "2f43ea69-ccc5-476d-9d09-b981ff0e1838"). InnerVolumeSpecName "kube-api-access-m74hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:33:10.574520 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.574501    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f43ea69-ccc5-476d-9d09-b981ff0e1838" (UID: "2f43ea69-ccc5-476d-9d09-b981ff0e1838"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:33:10.574520 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.574511    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-dshm" (OuterVolumeSpecName: "dshm") pod "2f43ea69-ccc5-476d-9d09-b981ff0e1838" (UID: "2f43ea69-ccc5-476d-9d09-b981ff0e1838"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:33:10.673556 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.673497    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:10.673556 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.673518    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43ea69-ccc5-476d-9d09-b981ff0e1838-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:10.673556 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.673527    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m74hj\" (UniqueName: \"kubernetes.io/projected/2f43ea69-ccc5-476d-9d09-b981ff0e1838-kube-api-access-m74hj\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:10.673556 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.673536    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43ea69-ccc5-476d-9d09-b981ff0e1838-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:33:10.847920 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.847892    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"]
Aug 01 01:33:10.849582 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:10.849560    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-65669d/rate-test-model-65669d-kserve-784655f48c-9772d"]
Aug 01 01:33:12.262980 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:12.262924    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f43ea69-ccc5-476d-9d09-b981ff0e1838" path="/var/lib/kubelet/pods/2f43ea69-ccc5-476d-9d09-b981ff0e1838/volumes"
Aug 01 01:33:18.830792 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.830753    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"]
Aug 01 01:33:18.831388 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.831368    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f43ea69-ccc5-476d-9d09-b981ff0e1838" containerName="main"
Aug 01 01:33:18.831458 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.831392    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f43ea69-ccc5-476d-9d09-b981ff0e1838" containerName="main"
Aug 01 01:33:18.831458 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.831414    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f10c774-47cc-4e2d-9229-5d264514eaef" containerName="main"
Aug 01 01:33:18.831458 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.831423    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f10c774-47cc-4e2d-9229-5d264514eaef" containerName="main"
Aug 01 01:33:18.831556 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.831522    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f10c774-47cc-4e2d-9229-5d264514eaef" containerName="main"
Aug 01 01:33:18.831556 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.831537    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f43ea69-ccc5-476d-9d09-b981ff0e1838" containerName="main"
Aug 01 01:33:18.836249 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.836226    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:18.838598 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.838577    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-de9aaa-serving-cert\""
Aug 01 01:33:18.845447 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.844450    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"]
Aug 01 01:33:18.847238 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.847212    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:18.847368 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.847264    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2q4w\" (UniqueName: \"kubernetes.io/projected/b34d77fd-dba4-42ba-9e14-d18c414d5d95-kube-api-access-m2q4w\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:18.847440 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.847420    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d77fd-dba4-42ba-9e14-d18c414d5d95-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:18.948666 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.948632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:18.948836 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.948676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2q4w\" (UniqueName: \"kubernetes.io/projected/b34d77fd-dba4-42ba-9e14-d18c414d5d95-kube-api-access-m2q4w\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:18.948836 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.948740    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d77fd-dba4-42ba-9e14-d18c414d5d95-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:18.948836 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:33:18.948794    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-serving-cert: secret "maas-api-e2e-ipp-a-de9aaa-serving-cert" not found
Aug 01 01:33:18.948986 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:33:18.948880    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls podName:b34d77fd-dba4-42ba-9e14-d18c414d5d95 nodeName:}" failed. No retries permitted until 2026-08-01 01:33:19.448860087 +0000 UTC m=+3565.682792330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls") pod "maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x" (UID: "b34d77fd-dba4-42ba-9e14-d18c414d5d95") : secret "maas-api-e2e-ipp-a-de9aaa-serving-cert" not found
Aug 01 01:33:18.949389 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.949372    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d77fd-dba4-42ba-9e14-d18c414d5d95-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:18.956760 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:18.956741    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2q4w\" (UniqueName: \"kubernetes.io/projected/b34d77fd-dba4-42ba-9e14-d18c414d5d95-kube-api-access-m2q4w\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:19.454637 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:19.454601    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:19.454838 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:33:19.454762    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-serving-cert: secret "maas-api-e2e-ipp-a-de9aaa-serving-cert" not found
Aug 01 01:33:19.454915 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:33:19.454852    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls podName:b34d77fd-dba4-42ba-9e14-d18c414d5d95 nodeName:}" failed. No retries permitted until 2026-08-01 01:33:20.454829714 +0000 UTC m=+3566.688761962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls") pod "maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x" (UID: "b34d77fd-dba4-42ba-9e14-d18c414d5d95") : secret "maas-api-e2e-ipp-a-de9aaa-serving-cert" not found
Aug 01 01:33:20.464785 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:20.464746    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:20.467241 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:20.467220    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:20.649307 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:20.649276    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:20.784387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:20.784361    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"]
Aug 01 01:33:20.786457 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:33:20.786431    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34d77fd_dba4_42ba_9e14_d18c414d5d95.slice/crio-03b6b90c3ea27a3bcedcb3ce6dfcf2ff00bdbb5e3eb23d93a3416d346dd6bf9d WatchSource:0}: Error finding container 03b6b90c3ea27a3bcedcb3ce6dfcf2ff00bdbb5e3eb23d93a3416d346dd6bf9d: Status 404 returned error can't find the container with id 03b6b90c3ea27a3bcedcb3ce6dfcf2ff00bdbb5e3eb23d93a3416d346dd6bf9d
Aug 01 01:33:20.787879 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:20.787860    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 01:33:21.581516 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:21.581476    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x" event={"ID":"b34d77fd-dba4-42ba-9e14-d18c414d5d95","Type":"ContainerStarted","Data":"2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c"}
Aug 01 01:33:21.582019 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:21.581538    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x" event={"ID":"b34d77fd-dba4-42ba-9e14-d18c414d5d95","Type":"ContainerStarted","Data":"03b6b90c3ea27a3bcedcb3ce6dfcf2ff00bdbb5e3eb23d93a3416d346dd6bf9d"}
Aug 01 01:33:21.582019 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:21.581588    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:27.591596 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:27.591564    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:33:27.607342 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:27.607288    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x" podStartSLOduration=9.109867978 podStartE2EDuration="9.60727054s" totalImagesPullingTime="497.402562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:33:21.597040967 +0000 UTC m=+3567.830973234" watchObservedRunningTime="2026-08-01 01:33:27.60727054 +0000 UTC m=+3573.841202808"
Aug 01 01:33:43.875345 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.875302    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"]
Aug 01 01:33:43.879038 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.879017    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.881533 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.881514    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9ba870-openshift-default-dockercfg-xb4lx\""
Aug 01 01:33:43.891029 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.891007    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"]
Aug 01 01:33:43.961854 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.961822    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-token\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.962018 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.961866    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-certs\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.962018 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.961883    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-podinfo\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.962018 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.961899    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istiod-ca-cert\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.962018 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.961983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxnj4\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-kube-api-access-mxnj4\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.962167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.962015    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-socket\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.962167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.962035    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-envoy\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.962167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.962069    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-data\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:43.962167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:43.962099    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-credential-socket\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063017 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.062983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-token\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063017 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063028    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-certs\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063046    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-podinfo\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063063    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istiod-ca-cert\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063086    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxnj4\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-kube-api-access-mxnj4\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-socket\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063305    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-envoy\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063603 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063327    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-data\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063603 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063377    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-credential-socket\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063603 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063501    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-certs\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063778 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063622    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-socket\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063778 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063716    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-data\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063894 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063824    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-credential-socket\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.063962 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.063912    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istiod-ca-cert\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.065565 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.065543    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-envoy\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.065848 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.065833    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-podinfo\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.071014 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.070995    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-token\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.071277 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.071259    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxnj4\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-kube-api-access-mxnj4\") pod \"e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") " pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.190997 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.190903    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:44.327066 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.327040    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"]
Aug 01 01:33:44.328905 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:33:44.328878    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c6ca43_6a2b_46f7_a8c9_889c1bc4b067.slice/crio-a6597e56e741ae2a423a32c8b1093752af0ec7db9a23647458a25ba3c08fcbbe WatchSource:0}: Error finding container a6597e56e741ae2a423a32c8b1093752af0ec7db9a23647458a25ba3c08fcbbe: Status 404 returned error can't find the container with id a6597e56e741ae2a423a32c8b1093752af0ec7db9a23647458a25ba3c08fcbbe
Aug 01 01:33:44.336557 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.331716    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:33:44.336557 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.331878    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:33:44.336557 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.331981    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:33:44.699652 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.699612    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" event={"ID":"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067","Type":"ContainerStarted","Data":"3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce"}
Aug 01 01:33:44.699831 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.699707    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" event={"ID":"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067","Type":"ContainerStarted","Data":"a6597e56e741ae2a423a32c8b1093752af0ec7db9a23647458a25ba3c08fcbbe"}
Aug 01 01:33:44.720571 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:44.720516    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" podStartSLOduration=1.720496537 podStartE2EDuration="1.720496537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:33:44.717119968 +0000 UTC m=+3590.951052235" watchObservedRunningTime="2026-08-01 01:33:44.720496537 +0000 UTC m=+3590.954428804"
Aug 01 01:33:45.191057 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:45.191021    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:45.196118 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:45.196092    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:45.705138 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:45.705098    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:33:45.706179 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:33:45.706154    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:34:00.706702 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:00.706610    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Aug 01 01:34:00.706702 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:00.706681    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Aug 01 01:34:00.785975 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:00.785910    2570 generic.go:356] "Generic (PLEG): container finished" podID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerID="3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce" exitCode=0
Aug 01 01:34:00.786167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:00.785992    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" event={"ID":"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067","Type":"ContainerDied","Data":"3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce"}
Aug 01 01:34:00.786469 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:00.786452    2570 scope.go:117] "RemoveContainer" containerID="3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce"
Aug 01 01:34:00.787620 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:00.787591    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:34:00.787708 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:00.787683    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:34:00.787748 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:00.787716    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 01:34:01.793679 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:01.793641    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" event={"ID":"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067","Type":"ContainerStarted","Data":"352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3"}
Aug 01 01:34:02.191484 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:02.191445    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:34:02.191758 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:02.191734    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Aug 01 01:34:02.191813 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:02.191784    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Aug 01 01:34:03.191563 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:03.191527    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Aug 01 01:34:03.192014 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:03.191600    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Aug 01 01:34:04.191431 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:04.191394    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Aug 01 01:34:04.191591 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:04.191466    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Aug 01 01:34:05.191266 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:05.191229    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Aug 01 01:34:05.191439 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:05.191287    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Aug 01 01:34:06.191605 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:06.191565    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Aug 01 01:34:06.192048 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:06.191628    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Aug 01 01:34:07.195645 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:07.195613    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:34:07.196110 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:07.195913    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:34:07.196517 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:07.196501    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:34:21.318127 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.318093    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"]
Aug 01 01:34:21.321711 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.321693    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.324790 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.324766    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-de9aaa\"/\"kube-root-ca.crt\""
Aug 01 01:34:21.325845 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.325824    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-de9aaa\"/\"default-dockercfg-85mm2\""
Aug 01 01:34:21.325985 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.325824    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-de9aaa\"/\"openshift-service-ca.crt\""
Aug 01 01:34:21.326069 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.326000    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-de9aaa\"/\"ipp-route-de9aaa-kserve-self-signed-certs\""
Aug 01 01:34:21.334581 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.334561    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"]
Aug 01 01:34:21.418286 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.418257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-home\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.418414 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.418327    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-dshm\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.418414 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.418359    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-model-cache\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.418414 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.418384    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcb64\" (UniqueName: \"kubernetes.io/projected/3fe27107-23af-43e0-8626-0c90b439f195-kube-api-access-mcb64\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.418524 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.418436    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe27107-23af-43e0-8626-0c90b439f195-tls-certs\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.418524 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.418482    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-tmp-dir\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.519642 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.519604    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-tmp-dir\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.519815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.519650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-home\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.519815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.519702    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-dshm\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.519815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.519717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-model-cache\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.519815 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.519734    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcb64\" (UniqueName: \"kubernetes.io/projected/3fe27107-23af-43e0-8626-0c90b439f195-kube-api-access-mcb64\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.520020 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.519812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe27107-23af-43e0-8626-0c90b439f195-tls-certs\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.520123 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.520102    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-tmp-dir\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.520171 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.520149    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-model-cache\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.520219 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.520178    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-home\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.522143 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.522119    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-dshm\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.522434 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.522414    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe27107-23af-43e0-8626-0c90b439f195-tls-certs\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.527518 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.527496    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcb64\" (UniqueName: \"kubernetes.io/projected/3fe27107-23af-43e0-8626-0c90b439f195-kube-api-access-mcb64\") pod \"ipp-route-de9aaa-kserve-c755c9b4c-vr6d5\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") " pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.631619 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.631544    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:21.761542 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.761517    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"]
Aug 01 01:34:21.763142 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:34:21.763113    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fe27107_23af_43e0_8626_0c90b439f195.slice/crio-0dd96dbf4fb5177ae8adcc45208089ecfc63512b3859d73f55f94b363122bf2f WatchSource:0}: Error finding container 0dd96dbf4fb5177ae8adcc45208089ecfc63512b3859d73f55f94b363122bf2f: Status 404 returned error can't find the container with id 0dd96dbf4fb5177ae8adcc45208089ecfc63512b3859d73f55f94b363122bf2f
Aug 01 01:34:21.892908 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.892812    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5" event={"ID":"3fe27107-23af-43e0-8626-0c90b439f195","Type":"ContainerStarted","Data":"7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5"}
Aug 01 01:34:21.892908 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.892858    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5" event={"ID":"3fe27107-23af-43e0-8626-0c90b439f195","Type":"ContainerStarted","Data":"0dd96dbf4fb5177ae8adcc45208089ecfc63512b3859d73f55f94b363122bf2f"}
Aug 01 01:34:21.909842 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:21.909797    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5" podStartSLOduration=0.909782863 podStartE2EDuration="909.782863ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:34:21.907697447 +0000 UTC m=+3628.141629714" watchObservedRunningTime="2026-08-01 01:34:21.909782863 +0000 UTC m=+3628.143715128"
Aug 01 01:34:31.631945 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:31.631905    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:31.632424 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:31.631968    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:31.644555 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:31.644534    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:34:31.957091 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:34:31.957010    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:36:03.020017 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.019980    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"]
Aug 01 01:36:03.020546 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.020261    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x" podUID="b34d77fd-dba4-42ba-9e14-d18c414d5d95" containerName="maas-api" containerID="cri-o://2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c" gracePeriod=30
Aug 01 01:36:03.291091 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.291065    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:36:03.398345 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.398314    2570 generic.go:356] "Generic (PLEG): container finished" podID="b34d77fd-dba4-42ba-9e14-d18c414d5d95" containerID="2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c" exitCode=0
Aug 01 01:36:03.398501 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.398350    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x" event={"ID":"b34d77fd-dba4-42ba-9e14-d18c414d5d95","Type":"ContainerDied","Data":"2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c"}
Aug 01 01:36:03.398501 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.398371    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"
Aug 01 01:36:03.398501 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.398393    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x" event={"ID":"b34d77fd-dba4-42ba-9e14-d18c414d5d95","Type":"ContainerDied","Data":"03b6b90c3ea27a3bcedcb3ce6dfcf2ff00bdbb5e3eb23d93a3416d346dd6bf9d"}
Aug 01 01:36:03.398501 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.398416    2570 scope.go:117] "RemoveContainer" containerID="2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c"
Aug 01 01:36:03.407115 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.407095    2570 scope.go:117] "RemoveContainer" containerID="2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c"
Aug 01 01:36:03.407393 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:36:03.407369    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c\": container with ID starting with 2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c not found: ID does not exist" containerID="2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c"
Aug 01 01:36:03.407446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.407409    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c"} err="failed to get container status \"2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c\": rpc error: code = NotFound desc = could not find container \"2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c\": container with ID starting with 2a2e789c68084fed1968e784e50d6fdb05b2856d75eeb4cd2b4bced094fffd4c not found: ID does not exist"
Aug 01 01:36:03.434705 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.434682    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d77fd-dba4-42ba-9e14-d18c414d5d95-openshift-service-ca\") pod \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") "
Aug 01 01:36:03.434842 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.434750    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls\") pod \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") "
Aug 01 01:36:03.434842 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.434777    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2q4w\" (UniqueName: \"kubernetes.io/projected/b34d77fd-dba4-42ba-9e14-d18c414d5d95-kube-api-access-m2q4w\") pod \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\" (UID: \"b34d77fd-dba4-42ba-9e14-d18c414d5d95\") "
Aug 01 01:36:03.435100 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.435075    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34d77fd-dba4-42ba-9e14-d18c414d5d95-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b34d77fd-dba4-42ba-9e14-d18c414d5d95" (UID: "b34d77fd-dba4-42ba-9e14-d18c414d5d95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:36:03.436887 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.436865    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b34d77fd-dba4-42ba-9e14-d18c414d5d95" (UID: "b34d77fd-dba4-42ba-9e14-d18c414d5d95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:36:03.437061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.436919    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34d77fd-dba4-42ba-9e14-d18c414d5d95-kube-api-access-m2q4w" (OuterVolumeSpecName: "kube-api-access-m2q4w") pod "b34d77fd-dba4-42ba-9e14-d18c414d5d95" (UID: "b34d77fd-dba4-42ba-9e14-d18c414d5d95"). InnerVolumeSpecName "kube-api-access-m2q4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:36:03.536493 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.536408    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d77fd-dba4-42ba-9e14-d18c414d5d95-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:03.536493 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.536442    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2q4w\" (UniqueName: \"kubernetes.io/projected/b34d77fd-dba4-42ba-9e14-d18c414d5d95-kube-api-access-m2q4w\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:03.536493 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.536452    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d77fd-dba4-42ba-9e14-d18c414d5d95-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:03.720159 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.720127    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"]
Aug 01 01:36:03.723809 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:03.723785    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de9aaa-695779cdc7-p6s4x"]
Aug 01 01:36:04.262825 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:04.262790    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34d77fd-dba4-42ba-9e14-d18c414d5d95" path="/var/lib/kubelet/pods/b34d77fd-dba4-42ba-9e14-d18c414d5d95/volumes"
Aug 01 01:36:08.634338 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:08.634303    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"]
Aug 01 01:36:23.653882 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.653822    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5" podUID="3fe27107-23af-43e0-8626-0c90b439f195" containerName="main" containerID="cri-o://7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5" gracePeriod=45
Aug 01 01:36:23.908078 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.908021    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:36:23.912560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.912539    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcb64\" (UniqueName: \"kubernetes.io/projected/3fe27107-23af-43e0-8626-0c90b439f195-kube-api-access-mcb64\") pod \"3fe27107-23af-43e0-8626-0c90b439f195\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") "
Aug 01 01:36:23.912614 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.912605    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-home\") pod \"3fe27107-23af-43e0-8626-0c90b439f195\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") "
Aug 01 01:36:23.912680 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.912670    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-tmp-dir\") pod \"3fe27107-23af-43e0-8626-0c90b439f195\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") "
Aug 01 01:36:23.912721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.912692    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe27107-23af-43e0-8626-0c90b439f195-tls-certs\") pod \"3fe27107-23af-43e0-8626-0c90b439f195\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") "
Aug 01 01:36:23.912721 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.912711    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-model-cache\") pod \"3fe27107-23af-43e0-8626-0c90b439f195\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") "
Aug 01 01:36:23.912822 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.912742    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-dshm\") pod \"3fe27107-23af-43e0-8626-0c90b439f195\" (UID: \"3fe27107-23af-43e0-8626-0c90b439f195\") "
Aug 01 01:36:23.912878 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.912859    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-home" (OuterVolumeSpecName: "home") pod "3fe27107-23af-43e0-8626-0c90b439f195" (UID: "3fe27107-23af-43e0-8626-0c90b439f195"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:36:23.913084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.913032    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fe27107-23af-43e0-8626-0c90b439f195" (UID: "3fe27107-23af-43e0-8626-0c90b439f195"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:36:23.913084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.913050    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fe27107-23af-43e0-8626-0c90b439f195" (UID: "3fe27107-23af-43e0-8626-0c90b439f195"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:36:23.913084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.913074    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-home\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:23.914841 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.914816    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe27107-23af-43e0-8626-0c90b439f195-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fe27107-23af-43e0-8626-0c90b439f195" (UID: "3fe27107-23af-43e0-8626-0c90b439f195"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:36:23.914912 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.914822    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe27107-23af-43e0-8626-0c90b439f195-kube-api-access-mcb64" (OuterVolumeSpecName: "kube-api-access-mcb64") pod "3fe27107-23af-43e0-8626-0c90b439f195" (UID: "3fe27107-23af-43e0-8626-0c90b439f195"). InnerVolumeSpecName "kube-api-access-mcb64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:36:23.914912 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:23.914889    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-dshm" (OuterVolumeSpecName: "dshm") pod "3fe27107-23af-43e0-8626-0c90b439f195" (UID: "3fe27107-23af-43e0-8626-0c90b439f195"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:36:24.013733 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.013700    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcb64\" (UniqueName: \"kubernetes.io/projected/3fe27107-23af-43e0-8626-0c90b439f195-kube-api-access-mcb64\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:24.013733 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.013729    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-tmp-dir\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:24.013733 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.013740    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe27107-23af-43e0-8626-0c90b439f195-tls-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:24.013974 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.013748    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-model-cache\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:24.013974 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.013757    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe27107-23af-43e0-8626-0c90b439f195-dshm\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:36:24.504209 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.504173    2570 generic.go:356] "Generic (PLEG): container finished" podID="3fe27107-23af-43e0-8626-0c90b439f195" containerID="7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5" exitCode=0
Aug 01 01:36:24.504393 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.504223    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"
Aug 01 01:36:24.504393 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.504258    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5" event={"ID":"3fe27107-23af-43e0-8626-0c90b439f195","Type":"ContainerDied","Data":"7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5"}
Aug 01 01:36:24.504393 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.504303    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5" event={"ID":"3fe27107-23af-43e0-8626-0c90b439f195","Type":"ContainerDied","Data":"0dd96dbf4fb5177ae8adcc45208089ecfc63512b3859d73f55f94b363122bf2f"}
Aug 01 01:36:24.504393 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.504320    2570 scope.go:117] "RemoveContainer" containerID="7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5"
Aug 01 01:36:24.517166 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.517141    2570 scope.go:117] "RemoveContainer" containerID="7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5"
Aug 01 01:36:24.518352 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:36:24.518322    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5\": container with ID starting with 7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5 not found: ID does not exist" containerID="7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5"
Aug 01 01:36:24.518560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.518535    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5"} err="failed to get container status \"7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5\": rpc error: code = NotFound desc = could not find container \"7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5\": container with ID starting with 7a807a5d58e67321c57418467034404385024d4de951e0957eb3c7a378409fc5 not found: ID does not exist"
Aug 01 01:36:24.520976 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.520955    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"]
Aug 01 01:36:24.525963 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:24.525902    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-de9aaa/ipp-route-de9aaa-kserve-c755c9b4c-vr6d5"]
Aug 01 01:36:26.262020 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:36:26.261984    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe27107-23af-43e0-8626-0c90b439f195" path="/var/lib/kubelet/pods/3fe27107-23af-43e0-8626-0c90b439f195/volumes"
Aug 01 01:37:08.911636 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:08.911593    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"]
Aug 01 01:37:08.912144 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:08.911855    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy" containerID="cri-o://352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3" gracePeriod=30
Aug 01 01:37:14.166113 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.166087    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:37:14.346684 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346653    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-token\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.346855 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346705    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-data\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.346855 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346724    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-credential-socket\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.346855 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346749    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istiod-ca-cert\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.346855 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346776    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-certs\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.346855 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346810    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-envoy\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.346855 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346852    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-podinfo\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.347188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346892    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxnj4\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-kube-api-access-mxnj4\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.347188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.346950    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-socket\") pod \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\" (UID: \"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067\") "
Aug 01 01:37:14.347188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.347022    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:14.347188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.347110    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:14.347188 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.347157    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:14.347448 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.347301    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.347448 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.347319    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.347448 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.347333    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.347448 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.347376    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:14.347448 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.347406    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:37:14.349129 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.349100    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:37:14.349234 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.349180    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-kube-api-access-mxnj4" (OuterVolumeSpecName: "kube-api-access-mxnj4") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "kube-api-access-mxnj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:14.349325 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.349308    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:14.349369 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.349318    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" (UID: "a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:14.448725 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.448693    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.448725 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.448719    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.448725 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.448731    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.448964 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.448743    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxnj4\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-kube-api-access-mxnj4\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.448964 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.448755    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.448964 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.448766    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:14.752037 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.751939    2570 generic.go:356] "Generic (PLEG): container finished" podID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerID="352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3" exitCode=0
Aug 01 01:37:14.752037 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.751961    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" event={"ID":"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067","Type":"ContainerDied","Data":"352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3"}
Aug 01 01:37:14.752037 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.751983    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"
Aug 01 01:37:14.752037 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.752020    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n" event={"ID":"a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067","Type":"ContainerDied","Data":"a6597e56e741ae2a423a32c8b1093752af0ec7db9a23647458a25ba3c08fcbbe"}
Aug 01 01:37:14.752320 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.752048    2570 scope.go:117] "RemoveContainer" containerID="352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3"
Aug 01 01:37:14.762058 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.762041    2570 scope.go:117] "RemoveContainer" containerID="3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce"
Aug 01 01:37:14.769827 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.769812    2570 scope.go:117] "RemoveContainer" containerID="352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3"
Aug 01 01:37:14.770089 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:14.770070    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3\": container with ID starting with 352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3 not found: ID does not exist" containerID="352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3"
Aug 01 01:37:14.770153 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.770098    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3"} err="failed to get container status \"352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3\": rpc error: code = NotFound desc = could not find container \"352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3\": container with ID starting with 352d315c15d885f3be19c8dedbfc0d2fc9620f8ada816addb863a04d2b65a8f3 not found: ID does not exist"
Aug 01 01:37:14.770153 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.770116    2570 scope.go:117] "RemoveContainer" containerID="3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce"
Aug 01 01:37:14.770335 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:14.770316    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce\": container with ID starting with 3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce not found: ID does not exist" containerID="3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce"
Aug 01 01:37:14.770388 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.770343    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce"} err="failed to get container status \"3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce\": rpc error: code = NotFound desc = could not find container \"3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce\": container with ID starting with 3975b218abd77a33958e16d2f9577aa6dd014d412acda0cbf784a849b2360bce not found: ID does not exist"
Aug 01 01:37:14.774273 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.774250    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"]
Aug 01 01:37:14.778090 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:14.778069    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ba870-openshift-default-6d4dfd6d86-fs67n"]
Aug 01 01:37:16.262091 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:16.262048    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" path="/var/lib/kubelet/pods/a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067/volumes"
Aug 01 01:37:19.880921 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:19.880886    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"]
Aug 01 01:37:19.881370 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:19.881167    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f" podUID="147bcadf-ee13-4c01-a561-0946523c5445" containerName="payload-processing" containerID="cri-o://51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd" gracePeriod=30
Aug 01 01:37:19.888223 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:19.888197    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"]
Aug 01 01:37:19.888443 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:19.888414    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5" podUID="8154cdf2-3ec9-4602-bdb9-fe7e60b5a570" containerName="payload-pre-processing" containerID="cri-o://377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399" gracePeriod=30
Aug 01 01:37:20.136965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.136869    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:37:20.159798 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.159773    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:37:20.196527 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.196499    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nmhp\" (UniqueName: \"kubernetes.io/projected/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-kube-api-access-4nmhp\") pod \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\" (UID: \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\") "
Aug 01 01:37:20.196693 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.196536    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-plugins-config-volume\") pod \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\" (UID: \"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570\") "
Aug 01 01:37:20.196693 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.196586    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147bcadf-ee13-4c01-a561-0946523c5445-plugins-config-volume\") pod \"147bcadf-ee13-4c01-a561-0946523c5445\" (UID: \"147bcadf-ee13-4c01-a561-0946523c5445\") "
Aug 01 01:37:20.196693 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.196631    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/147bcadf-ee13-4c01-a561-0946523c5445-kube-api-access-g9j8n\") pod \"147bcadf-ee13-4c01-a561-0946523c5445\" (UID: \"147bcadf-ee13-4c01-a561-0946523c5445\") "
Aug 01 01:37:20.197030 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.197009    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8154cdf2-3ec9-4602-bdb9-fe7e60b5a570" (UID: "8154cdf2-3ec9-4602-bdb9-fe7e60b5a570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:37:20.197145 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.197036    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147bcadf-ee13-4c01-a561-0946523c5445-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "147bcadf-ee13-4c01-a561-0946523c5445" (UID: "147bcadf-ee13-4c01-a561-0946523c5445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:37:20.198612 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.198578    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-kube-api-access-4nmhp" (OuterVolumeSpecName: "kube-api-access-4nmhp") pod "8154cdf2-3ec9-4602-bdb9-fe7e60b5a570" (UID: "8154cdf2-3ec9-4602-bdb9-fe7e60b5a570"). InnerVolumeSpecName "kube-api-access-4nmhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:20.198765 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.198744    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147bcadf-ee13-4c01-a561-0946523c5445-kube-api-access-g9j8n" (OuterVolumeSpecName: "kube-api-access-g9j8n") pod "147bcadf-ee13-4c01-a561-0946523c5445" (UID: "147bcadf-ee13-4c01-a561-0946523c5445"). InnerVolumeSpecName "kube-api-access-g9j8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:20.297764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.297735    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/147bcadf-ee13-4c01-a561-0946523c5445-kube-api-access-g9j8n\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:20.297764 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.297759    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmhp\" (UniqueName: \"kubernetes.io/projected/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-kube-api-access-4nmhp\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:20.297912 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.297768    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:20.297912 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.297778    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147bcadf-ee13-4c01-a561-0946523c5445-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:20.781992 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.781958    2570 generic.go:356] "Generic (PLEG): container finished" podID="147bcadf-ee13-4c01-a561-0946523c5445" containerID="51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd" exitCode=0
Aug 01 01:37:20.782167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.782005    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"
Aug 01 01:37:20.782167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.782039    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f" event={"ID":"147bcadf-ee13-4c01-a561-0946523c5445","Type":"ContainerDied","Data":"51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd"}
Aug 01 01:37:20.782167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.782088    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f" event={"ID":"147bcadf-ee13-4c01-a561-0946523c5445","Type":"ContainerDied","Data":"4278c9d87a3119b0cd0600214dc526ee698b452c27207e607924328d6c815c6e"}
Aug 01 01:37:20.782167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.782111    2570 scope.go:117] "RemoveContainer" containerID="51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd"
Aug 01 01:37:20.783424 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.783401    2570 generic.go:356] "Generic (PLEG): container finished" podID="8154cdf2-3ec9-4602-bdb9-fe7e60b5a570" containerID="377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399" exitCode=0
Aug 01 01:37:20.783549 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.783439    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"
Aug 01 01:37:20.783549 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.783465    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5" event={"ID":"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570","Type":"ContainerDied","Data":"377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399"}
Aug 01 01:37:20.783549 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.783499    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5" event={"ID":"8154cdf2-3ec9-4602-bdb9-fe7e60b5a570","Type":"ContainerDied","Data":"ef9dab88b84515f31888885de21bcf1b685928e1445aff4196d06a9cb302ab47"}
Aug 01 01:37:20.793484 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.793466    2570 scope.go:117] "RemoveContainer" containerID="51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd"
Aug 01 01:37:20.793763 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:20.793744    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd\": container with ID starting with 51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd not found: ID does not exist" containerID="51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd"
Aug 01 01:37:20.793826 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.793771    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd"} err="failed to get container status \"51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd\": rpc error: code = NotFound desc = could not find container \"51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd\": container with ID starting with 51de7808d1a0607cfb3325b6574727948f98f51c3a1ef86ef8ef2a7aa6628ebd not found: ID does not exist"
Aug 01 01:37:20.793826 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.793788    2570 scope.go:117] "RemoveContainer" containerID="377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399"
Aug 01 01:37:20.801528 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.801511    2570 scope.go:117] "RemoveContainer" containerID="377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399"
Aug 01 01:37:20.801772 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:20.801752    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399\": container with ID starting with 377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399 not found: ID does not exist" containerID="377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399"
Aug 01 01:37:20.801813 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.801781    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399"} err="failed to get container status \"377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399\": rpc error: code = NotFound desc = could not find container \"377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399\": container with ID starting with 377bbbdfc17d24ab3cc8035657f9c10f4099141c66d1a645db553a0a47367399 not found: ID does not exist"
Aug 01 01:37:20.802604 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.802585    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"]
Aug 01 01:37:20.809424 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.809400    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-423d15-5dd7db5796-bc6k5"]
Aug 01 01:37:20.819854 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.819833    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"]
Aug 01 01:37:20.825483 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:20.825459    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-423d15-77f7bd4478-6lb9f"]
Aug 01 01:37:22.262869 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:22.262835    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="147bcadf-ee13-4c01-a561-0946523c5445" path="/var/lib/kubelet/pods/147bcadf-ee13-4c01-a561-0946523c5445/volumes"
Aug 01 01:37:22.263291 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:22.263256    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8154cdf2-3ec9-4602-bdb9-fe7e60b5a570" path="/var/lib/kubelet/pods/8154cdf2-3ec9-4602-bdb9-fe7e60b5a570/volumes"
Aug 01 01:37:26.472628 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:26.472589    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"]
Aug 01 01:37:26.473189 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:26.472824    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy" containerID="cri-o://283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3" gracePeriod=30
Aug 01 01:37:31.729131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.729107    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:37:31.787872 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.787794    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q545h\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-kube-api-access-q545h\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.787872 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.787845    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab752358-b918-4050-98df-3c17df616610-istio-podinfo\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.788131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.787875    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-data\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.788131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.787905    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab752358-b918-4050-98df-3c17df616610-istiod-ca-cert\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.788131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.787921    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-envoy\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.788131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.787972    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-socket\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.788131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788021    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-credential-socket\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.788131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788054    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-istio-token\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.788131 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788109    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-certs\") pod \"ab752358-b918-4050-98df-3c17df616610\" (UID: \"ab752358-b918-4050-98df-3c17df616610\") "
Aug 01 01:37:31.788514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788247    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:31.788514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788330    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:31.788514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788361    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:31.788514 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788451    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:31.788735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788556    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:31.788735 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.788562    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab752358-b918-4050-98df-3c17df616610-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:37:31.790348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.790316    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:31.790348 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.790333    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:31.790585 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.790562    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-kube-api-access-q545h" (OuterVolumeSpecName: "kube-api-access-q545h") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "kube-api-access-q545h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:31.790585 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.790576    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab752358-b918-4050-98df-3c17df616610-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab752358-b918-4050-98df-3c17df616610" (UID: "ab752358-b918-4050-98df-3c17df616610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:37:31.837437 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.837401    2570 generic.go:356] "Generic (PLEG): container finished" podID="ab752358-b918-4050-98df-3c17df616610" containerID="283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3" exitCode=0
Aug 01 01:37:31.837580 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.837459    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"
Aug 01 01:37:31.837580 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.837489    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" event={"ID":"ab752358-b918-4050-98df-3c17df616610","Type":"ContainerDied","Data":"283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3"}
Aug 01 01:37:31.837580 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.837543    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k" event={"ID":"ab752358-b918-4050-98df-3c17df616610","Type":"ContainerDied","Data":"610bb776bbeef5dc89c770a16cdd350375ec9c077fe2525709db399bdaaa96a8"}
Aug 01 01:37:31.837580 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.837570    2570 scope.go:117] "RemoveContainer" containerID="283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3"
Aug 01 01:37:31.846586 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.846568    2570 scope.go:117] "RemoveContainer" containerID="303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299"
Aug 01 01:37:31.854543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.854525    2570 scope.go:117] "RemoveContainer" containerID="283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3"
Aug 01 01:37:31.854871 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:31.854847    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3\": container with ID starting with 283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3 not found: ID does not exist" containerID="283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3"
Aug 01 01:37:31.854972 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.854882    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3"} err="failed to get container status \"283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3\": rpc error: code = NotFound desc = could not find container \"283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3\": container with ID starting with 283743c873a8b169f467aed92c5668f1b8a633cca96c75f7c009fc88cee7ddf3 not found: ID does not exist"
Aug 01 01:37:31.854972 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.854907    2570 scope.go:117] "RemoveContainer" containerID="303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299"
Aug 01 01:37:31.855198 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:31.855179    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299\": container with ID starting with 303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299 not found: ID does not exist" containerID="303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299"
Aug 01 01:37:31.855292 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.855209    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299"} err="failed to get container status \"303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299\": rpc error: code = NotFound desc = could not find container \"303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299\": container with ID starting with 303ae066678a555871e14dd9fc9260f260f497af3fd166ad6603a2e8d70ce299 not found: ID does not exist"
Aug 01 01:37:31.859654 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.859635    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"]
Aug 01 01:37:31.862866 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.862846    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-423d15-openshift-default-57f7fd8b48-kl76k"]
Aug 01 01:37:31.889605 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.889580    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q545h\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-kube-api-access-q545h\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:31.889710 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.889608    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab752358-b918-4050-98df-3c17df616610-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:31.889710 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.889622    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab752358-b918-4050-98df-3c17df616610-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:31.889710 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.889634    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:31.889710 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.889646    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:31.889710 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.889660    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:31.889710 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.889672    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab752358-b918-4050-98df-3c17df616610-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:31.889710 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:31.889686    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab752358-b918-4050-98df-3c17df616610-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:32.262607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:32.262573    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab752358-b918-4050-98df-3c17df616610" path="/var/lib/kubelet/pods/ab752358-b918-4050-98df-3c17df616610/volumes"
Aug 01 01:37:37.411004 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.410969    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"]
Aug 01 01:37:37.411437 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.411222    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj" podUID="9474041b-b9fb-4647-a03f-4b448599a5b8" containerName="maas-api" containerID="cri-o://4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c" gracePeriod=30
Aug 01 01:37:37.475359 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.475326    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"]
Aug 01 01:37:37.475622 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.475587    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk" podUID="eb36149e-d787-4487-9408-33265e84bdf4" containerName="payload-processing" containerID="cri-o://b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757" gracePeriod=30
Aug 01 01:37:37.727124 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.727100    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:37:37.760222 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.760198    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:37:37.838444 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.838408    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36149e-d787-4487-9408-33265e84bdf4-plugins-config-volume\") pod \"eb36149e-d787-4487-9408-33265e84bdf4\" (UID: \"eb36149e-d787-4487-9408-33265e84bdf4\") "
Aug 01 01:37:37.838617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.838470    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvjlq\" (UniqueName: \"kubernetes.io/projected/9474041b-b9fb-4647-a03f-4b448599a5b8-kube-api-access-vvjlq\") pod \"9474041b-b9fb-4647-a03f-4b448599a5b8\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") "
Aug 01 01:37:37.838617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.838498    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls\") pod \"9474041b-b9fb-4647-a03f-4b448599a5b8\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") "
Aug 01 01:37:37.838617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.838533    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9474041b-b9fb-4647-a03f-4b448599a5b8-openshift-service-ca\") pod \"9474041b-b9fb-4647-a03f-4b448599a5b8\" (UID: \"9474041b-b9fb-4647-a03f-4b448599a5b8\") "
Aug 01 01:37:37.838617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.838560    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtl5r\" (UniqueName: \"kubernetes.io/projected/eb36149e-d787-4487-9408-33265e84bdf4-kube-api-access-wtl5r\") pod \"eb36149e-d787-4487-9408-33265e84bdf4\" (UID: \"eb36149e-d787-4487-9408-33265e84bdf4\") "
Aug 01 01:37:37.838873 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.838847    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb36149e-d787-4487-9408-33265e84bdf4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb36149e-d787-4487-9408-33265e84bdf4" (UID: "eb36149e-d787-4487-9408-33265e84bdf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:37:37.838958 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.838908    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9474041b-b9fb-4647-a03f-4b448599a5b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9474041b-b9fb-4647-a03f-4b448599a5b8" (UID: "9474041b-b9fb-4647-a03f-4b448599a5b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:37:37.840670 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.840645    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9474041b-b9fb-4647-a03f-4b448599a5b8-kube-api-access-vvjlq" (OuterVolumeSpecName: "kube-api-access-vvjlq") pod "9474041b-b9fb-4647-a03f-4b448599a5b8" (UID: "9474041b-b9fb-4647-a03f-4b448599a5b8"). InnerVolumeSpecName "kube-api-access-vvjlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:37.840670 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.840659    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb36149e-d787-4487-9408-33265e84bdf4-kube-api-access-wtl5r" (OuterVolumeSpecName: "kube-api-access-wtl5r") pod "eb36149e-d787-4487-9408-33265e84bdf4" (UID: "eb36149e-d787-4487-9408-33265e84bdf4"). InnerVolumeSpecName "kube-api-access-wtl5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:37.840817 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.840693    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9474041b-b9fb-4647-a03f-4b448599a5b8" (UID: "9474041b-b9fb-4647-a03f-4b448599a5b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 01:37:37.867890 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.867863    2570 generic.go:356] "Generic (PLEG): container finished" podID="eb36149e-d787-4487-9408-33265e84bdf4" containerID="b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757" exitCode=0
Aug 01 01:37:37.867999 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.867915    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"
Aug 01 01:37:37.867999 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.867960    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk" event={"ID":"eb36149e-d787-4487-9408-33265e84bdf4","Type":"ContainerDied","Data":"b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757"}
Aug 01 01:37:37.868104 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.868012    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk" event={"ID":"eb36149e-d787-4487-9408-33265e84bdf4","Type":"ContainerDied","Data":"294653c050a7abc8c49d58f8a6c3851a13b39ed044262f95babdbb657b291207"}
Aug 01 01:37:37.868104 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.868041    2570 scope.go:117] "RemoveContainer" containerID="b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757"
Aug 01 01:37:37.869443 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.869415    2570 generic.go:356] "Generic (PLEG): container finished" podID="9474041b-b9fb-4647-a03f-4b448599a5b8" containerID="4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c" exitCode=0
Aug 01 01:37:37.869560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.869450    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj" event={"ID":"9474041b-b9fb-4647-a03f-4b448599a5b8","Type":"ContainerDied","Data":"4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c"}
Aug 01 01:37:37.869560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.869467    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"
Aug 01 01:37:37.869560 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.869496    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj" event={"ID":"9474041b-b9fb-4647-a03f-4b448599a5b8","Type":"ContainerDied","Data":"3b4472c16f4a403377b91858edbd41bd9abe4bceafe156bff7e2bc3b8ee59d91"}
Aug 01 01:37:37.877561 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.877521    2570 scope.go:117] "RemoveContainer" containerID="b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757"
Aug 01 01:37:37.877867 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:37.877843    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757\": container with ID starting with b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757 not found: ID does not exist" containerID="b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757"
Aug 01 01:37:37.877954 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.877875    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757"} err="failed to get container status \"b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757\": rpc error: code = NotFound desc = could not find container \"b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757\": container with ID starting with b3649bd82f52e9cabdbf35534dd640fef488254e23f60e9e5091f95d08da4757 not found: ID does not exist"
Aug 01 01:37:37.877954 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.877894    2570 scope.go:117] "RemoveContainer" containerID="4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c"
Aug 01 01:37:37.886187 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.886170    2570 scope.go:117] "RemoveContainer" containerID="4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c"
Aug 01 01:37:37.886422 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:37.886404    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c\": container with ID starting with 4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c not found: ID does not exist" containerID="4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c"
Aug 01 01:37:37.886473 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.886428    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c"} err="failed to get container status \"4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c\": rpc error: code = NotFound desc = could not find container \"4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c\": container with ID starting with 4a434b500380c42635d018270ac2bf1614299b603c8b7514c19193025535135c not found: ID does not exist"
Aug 01 01:37:37.893027 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.893007    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"]
Aug 01 01:37:37.897965 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.897943    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-65669d-69f4ff47f-hqzmj"]
Aug 01 01:37:37.908771 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.908749    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"]
Aug 01 01:37:37.912703 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.912681    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-65669d-9df7fcb54-xbnbk"]
Aug 01 01:37:37.939607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.939560    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36149e-d787-4487-9408-33265e84bdf4-plugins-config-volume\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:37.939607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.939580    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvjlq\" (UniqueName: \"kubernetes.io/projected/9474041b-b9fb-4647-a03f-4b448599a5b8-kube-api-access-vvjlq\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:37.939607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.939590    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9474041b-b9fb-4647-a03f-4b448599a5b8-maas-api-tls\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:37.939607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.939598    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9474041b-b9fb-4647-a03f-4b448599a5b8-openshift-service-ca\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:37.939607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:37.939607    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtl5r\" (UniqueName: \"kubernetes.io/projected/eb36149e-d787-4487-9408-33265e84bdf4-kube-api-access-wtl5r\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:38.261766 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:38.261691    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9474041b-b9fb-4647-a03f-4b448599a5b8" path="/var/lib/kubelet/pods/9474041b-b9fb-4647-a03f-4b448599a5b8/volumes"
Aug 01 01:37:38.262061 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:38.262046    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb36149e-d787-4487-9408-33265e84bdf4" path="/var/lib/kubelet/pods/eb36149e-d787-4487-9408-33265e84bdf4/volumes"
Aug 01 01:37:44.136371 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:44.136333    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"]
Aug 01 01:37:44.136857 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:44.136582    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy" containerID="cri-o://b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7" gracePeriod=30
Aug 01 01:37:44.876166 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:44.876130    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d78f965b-b2dv4_49034aed-cac8-43c0-bb75-8674a201d17e/authorino/0.log"
Aug 01 01:37:49.391692 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.386081    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:37:49.432307 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432224    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01c24-0eac-420c-abd0-bbf9912def63-istiod-ca-cert\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432307 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432275    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-envoy\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432307 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432292    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-data\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432553 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432415    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01c24-0eac-420c-abd0-bbf9912def63-istio-podinfo\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432553 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432450    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-certs\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432553 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432474    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-credential-socket\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432553 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432506    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-istio-token\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432759 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432558    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x9bz\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-kube-api-access-6x9bz\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432759 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432582    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-data" (OuterVolumeSpecName: "istio-data") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:49.432759 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432587    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-socket\") pod \"45c01c24-0eac-420c-abd0-bbf9912def63\" (UID: \"45c01c24-0eac-420c-abd0-bbf9912def63\") "
Aug 01 01:37:49.432759 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432654    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c01c24-0eac-420c-abd0-bbf9912def63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 01:37:49.433002 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432764    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:49.433002 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432777    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:49.433002 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.432973    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:49.433169 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.433009    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c01c24-0eac-420c-abd0-bbf9912def63-istiod-ca-cert\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.433169 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.433037    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-data\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.433169 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.433052    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-certs\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.433169 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.433064    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-workload-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.434729 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.434693    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-istio-token" (OuterVolumeSpecName: "istio-token") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:49.434729 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.434698    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45c01c24-0eac-420c-abd0-bbf9912def63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 01:37:49.434907 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.434708    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-kube-api-access-6x9bz" (OuterVolumeSpecName: "kube-api-access-6x9bz") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "kube-api-access-6x9bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 01:37:49.434907 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.434745    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45c01c24-0eac-420c-abd0-bbf9912def63" (UID: "45c01c24-0eac-420c-abd0-bbf9912def63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 01:37:49.533903 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.533875    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-istio-envoy\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.533903 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.533901    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c01c24-0eac-420c-abd0-bbf9912def63-istio-podinfo\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.534084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.533910    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c01c24-0eac-420c-abd0-bbf9912def63-credential-socket\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.534084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.533919    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-istio-token\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.534084 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.533940    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9bz\" (UniqueName: \"kubernetes.io/projected/45c01c24-0eac-420c-abd0-bbf9912def63-kube-api-access-6x9bz\") on node \"ip-10-0-141-142.ec2.internal\" DevicePath \"\""
Aug 01 01:37:49.931863 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.931828    2570 generic.go:356] "Generic (PLEG): container finished" podID="45c01c24-0eac-420c-abd0-bbf9912def63" containerID="b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7" exitCode=0
Aug 01 01:37:49.932045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.931890    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" event={"ID":"45c01c24-0eac-420c-abd0-bbf9912def63","Type":"ContainerDied","Data":"b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7"}
Aug 01 01:37:49.932045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.931896    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"
Aug 01 01:37:49.932045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.931980    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf" event={"ID":"45c01c24-0eac-420c-abd0-bbf9912def63","Type":"ContainerDied","Data":"bdd32497ef4dea8949be3830611f3912fd09e087214b6b1ddb9aa2e79dc83522"}
Aug 01 01:37:49.932045 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.932005    2570 scope.go:117] "RemoveContainer" containerID="b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7"
Aug 01 01:37:49.940967 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.940909    2570 scope.go:117] "RemoveContainer" containerID="6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09"
Aug 01 01:37:49.949398 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.949381    2570 scope.go:117] "RemoveContainer" containerID="b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7"
Aug 01 01:37:49.949685 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:49.949664    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7\": container with ID starting with b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7 not found: ID does not exist" containerID="b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7"
Aug 01 01:37:49.949733 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.949694    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7"} err="failed to get container status \"b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7\": rpc error: code = NotFound desc = could not find container \"b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7\": container with ID starting with b2f0009b18830d6f5001d75aa78af4348666c473f239c4cf5ed70db3bc0cc5a7 not found: ID does not exist"
Aug 01 01:37:49.949733 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.949711    2570 scope.go:117] "RemoveContainer" containerID="6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09"
Aug 01 01:37:49.949965 ip-10-0-141-142 kubenswrapper[2570]: E0801 01:37:49.949926    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09\": container with ID starting with 6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09 not found: ID does not exist" containerID="6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09"
Aug 01 01:37:49.950032 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.949975    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09"} err="failed to get container status \"6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09\": rpc error: code = NotFound desc = could not find container \"6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09\": container with ID starting with 6e90b8c03130b5b6e7f38e98ea98c246d5cdf6ab6121acc66d305073ada46a09 not found: ID does not exist"
Aug 01 01:37:49.953420 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.953394    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"]
Aug 01 01:37:49.958883 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:49.958864    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-65669d-openshift-default-8f4884544-jczpf"]
Aug 01 01:37:50.262554 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:50.262464    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" path="/var/lib/kubelet/pods/45c01c24-0eac-420c-abd0-bbf9912def63/volumes"
Aug 01 01:37:50.994943 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:50.994911    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787d5bc65-p4m2m_5778d33b-8872-4191-96ab-c210256d3c25/manager/0.log"
Aug 01 01:37:51.121453 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:51.121425    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2npxt_7f8797bf-066c-4c26-9ffd-5d1382eee2b5/server/0.log"
Aug 01 01:37:51.491281 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:51.491252    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f88dcc48-cm5hv_5c2ef3b9-05f2-47bb-ac1b-6f64858c0874/copy-manifests/0.log"
Aug 01 01:37:51.504444 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:51.504420    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f88dcc48-cm5hv_5c2ef3b9-05f2-47bb-ac1b-6f64858c0874/manager/0.log"
Aug 01 01:37:52.033660 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:52.033630    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8696875bbd-bnc6p_40b2c1c2-f74a-412f-b796-2f0e80d5c22a/maas-api/0.log"
Aug 01 01:37:52.156422 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:52.156394    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29759100-68zjx_b011ff59-9da2-448e-9350-98b7e1b4699f/cleanup/0.log"
Aug 01 01:37:52.302846 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:52.302768    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29759115-sbjlf_816d33c7-b1e0-4de7-a170-824b1e3b9cdd/cleanup/0.log"
Aug 01 01:37:52.543032 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:52.542990    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-bfvwf_2a9e6955-3ad0-4eb5-876b-2ed51efc036d/postgres/0.log"
Aug 01 01:37:53.410723 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:53.410692    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d78f965b-b2dv4_49034aed-cac8-43c0-bb75-8674a201d17e/authorino/0.log"
Aug 01 01:37:54.829969 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:54.829910    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7798dfdbd7-l7cgg_4212e104-5c9d-49b3-8fca-3e69f1413f3a/kube-auth-proxy/0.log"
Aug 01 01:37:55.222597 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:55.222569    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-ld4rx_3020d525-e75f-4340-b6c2-e28a56c409b4/payload-pre-processing/0.log"
Aug 01 01:37:56.021996 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:37:56.021966    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dxzgn_eacbf814-ce99-403e-84d8-756772bcfd53/main/0.log"
Aug 01 01:38:02.519385 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519347    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jdcjl/must-gather-4kdfr"]
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519718    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9474041b-b9fb-4647-a03f-4b448599a5b8" containerName="maas-api"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519731    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9474041b-b9fb-4647-a03f-4b448599a5b8" containerName="maas-api"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519742    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519751    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519764    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519771    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519778    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147bcadf-ee13-4c01-a561-0946523c5445" containerName="payload-processing"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519784    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="147bcadf-ee13-4c01-a561-0946523c5445" containerName="payload-processing"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519794    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34d77fd-dba4-42ba-9e14-d18c414d5d95" containerName="maas-api"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519799    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34d77fd-dba4-42ba-9e14-d18c414d5d95" containerName="maas-api"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519813    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb36149e-d787-4487-9408-33265e84bdf4" containerName="payload-processing"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519820    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb36149e-d787-4487-9408-33265e84bdf4" containerName="payload-processing"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519832    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519839    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy"
Aug 01 01:38:02.519839 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519846    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519852    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519859    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8154cdf2-3ec9-4602-bdb9-fe7e60b5a570" containerName="payload-pre-processing"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519864    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8154cdf2-3ec9-4602-bdb9-fe7e60b5a570" containerName="payload-pre-processing"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519876    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe27107-23af-43e0-8626-0c90b439f195" containerName="main"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519881    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe27107-23af-43e0-8626-0c90b439f195" containerName="main"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519945    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34d77fd-dba4-42ba-9e14-d18c414d5d95" containerName="maas-api"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519953    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb36149e-d787-4487-9408-33265e84bdf4" containerName="payload-processing"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519960    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9474041b-b9fb-4647-a03f-4b448599a5b8" containerName="maas-api"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519966    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519972    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519977    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519982    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c01c24-0eac-420c-abd0-bbf9912def63" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519987    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519993    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="147bcadf-ee13-4c01-a561-0946523c5445" containerName="payload-processing"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.519998    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.520004    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8154cdf2-3ec9-4602-bdb9-fe7e60b5a570" containerName="payload-pre-processing"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.520009    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe27107-23af-43e0-8626-0c90b439f195" containerName="main"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.520054    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.520060    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c6ca43-6a2b-46f7-a8c9-889c1bc4b067" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.520068    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy"
Aug 01 01:38:02.520436 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.520073    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab752358-b918-4050-98df-3c17df616610" containerName="istio-proxy"
Aug 01 01:38:02.524912 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.524893    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdcjl/must-gather-4kdfr"
Aug 01 01:38:02.527339 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.527309    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jdcjl\"/\"kube-root-ca.crt\""
Aug 01 01:38:02.528331 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.528307    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jdcjl\"/\"default-dockercfg-dsx7t\""
Aug 01 01:38:02.528441 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.528340    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jdcjl\"/\"openshift-service-ca.crt\""
Aug 01 01:38:02.533808 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.533785    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdcjl/must-gather-4kdfr"]
Aug 01 01:38:02.638200 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.638150    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54873702-79a6-4818-bf8a-a357a358eb95-must-gather-output\") pod \"must-gather-4kdfr\" (UID: \"54873702-79a6-4818-bf8a-a357a358eb95\") " pod="openshift-must-gather-jdcjl/must-gather-4kdfr"
Aug 01 01:38:02.638387 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.638249    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgggx\" (UniqueName: \"kubernetes.io/projected/54873702-79a6-4818-bf8a-a357a358eb95-kube-api-access-tgggx\") pod \"must-gather-4kdfr\" (UID: \"54873702-79a6-4818-bf8a-a357a358eb95\") " pod="openshift-must-gather-jdcjl/must-gather-4kdfr"
Aug 01 01:38:02.738678 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.738640    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54873702-79a6-4818-bf8a-a357a358eb95-must-gather-output\") pod \"must-gather-4kdfr\" (UID: \"54873702-79a6-4818-bf8a-a357a358eb95\") " pod="openshift-must-gather-jdcjl/must-gather-4kdfr"
Aug 01 01:38:02.738878 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.738698    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgggx\" (UniqueName: \"kubernetes.io/projected/54873702-79a6-4818-bf8a-a357a358eb95-kube-api-access-tgggx\") pod \"must-gather-4kdfr\" (UID: \"54873702-79a6-4818-bf8a-a357a358eb95\") " pod="openshift-must-gather-jdcjl/must-gather-4kdfr"
Aug 01 01:38:02.739078 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.739056    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54873702-79a6-4818-bf8a-a357a358eb95-must-gather-output\") pod \"must-gather-4kdfr\" (UID: \"54873702-79a6-4818-bf8a-a357a358eb95\") " pod="openshift-must-gather-jdcjl/must-gather-4kdfr"
Aug 01 01:38:02.746507 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.746484    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgggx\" (UniqueName: \"kubernetes.io/projected/54873702-79a6-4818-bf8a-a357a358eb95-kube-api-access-tgggx\") pod \"must-gather-4kdfr\" (UID: \"54873702-79a6-4818-bf8a-a357a358eb95\") " pod="openshift-must-gather-jdcjl/must-gather-4kdfr"
Aug 01 01:38:02.835523 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.835490    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdcjl/must-gather-4kdfr"
Aug 01 01:38:02.966418 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.966285    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdcjl/must-gather-4kdfr"]
Aug 01 01:38:02.969092 ip-10-0-141-142 kubenswrapper[2570]: W0801 01:38:02.969064    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54873702_79a6_4818_bf8a_a357a358eb95.slice/crio-30df98f7e83e6a93cac2f773489e432a1c57b8943053d72c2dedabc332f9f395 WatchSource:0}: Error finding container 30df98f7e83e6a93cac2f773489e432a1c57b8943053d72c2dedabc332f9f395: Status 404 returned error can't find the container with id 30df98f7e83e6a93cac2f773489e432a1c57b8943053d72c2dedabc332f9f395
Aug 01 01:38:02.992625 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:02.992595    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdcjl/must-gather-4kdfr" event={"ID":"54873702-79a6-4818-bf8a-a357a358eb95","Type":"ContainerStarted","Data":"30df98f7e83e6a93cac2f773489e432a1c57b8943053d72c2dedabc332f9f395"}
Aug 01 01:38:04.000484 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:04.000440    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdcjl/must-gather-4kdfr" event={"ID":"54873702-79a6-4818-bf8a-a357a358eb95","Type":"ContainerStarted","Data":"d65abfa2292bae9a3f98504226691499a07831b9c7c1fc6463cff444068c7126"}
Aug 01 01:38:05.008117 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:05.008072    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdcjl/must-gather-4kdfr" event={"ID":"54873702-79a6-4818-bf8a-a357a358eb95","Type":"ContainerStarted","Data":"6f992749db9e57389313667af43769408b1ec6c2324d554e0dc8837a31d333ba"}
Aug 01 01:38:05.026690 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:05.026626    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jdcjl/must-gather-4kdfr" podStartSLOduration=2.176941338 podStartE2EDuration="3.026610312s" totalImagesPullingTime="849.668974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:38:05.024017706 +0000 UTC m=+3851.257949972" watchObservedRunningTime="2026-08-01 01:38:05.026610312 +0000 UTC m=+3851.260542577"
Aug 01 01:38:05.366501 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:05.366466    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pgtlh_f5c55ca5-8428-4fbf-acbf-5c3d996c68d5/global-pull-secret-syncer/0.log"
Aug 01 01:38:05.518761 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:05.518725    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wk8l6_9c68d149-59f9-4989-b7da-ae32080c74ba/konnectivity-agent/0.log"
Aug 01 01:38:05.581277 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:05.581246    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-142.ec2.internal_f923c99785cedeeb9d5a1e0081b46a58/haproxy/0.log"
Aug 01 01:38:09.837216 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:09.837157    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d78f965b-b2dv4_49034aed-cac8-43c0-bb75-8674a201d17e/authorino/0.log"
Aug 01 01:38:11.928195 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:11.928168    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d789d7b79-t6xcx_d6de4a72-b91c-4031-9fd4-6dd92099ec54/metrics-server/0.log"
Aug 01 01:38:12.064801 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.064773    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t58fj_8d0f282b-d478-4315-b903-74f767be07ae/node-exporter/0.log"
Aug 01 01:38:12.087167 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.087138    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t58fj_8d0f282b-d478-4315-b903-74f767be07ae/kube-rbac-proxy/0.log"
Aug 01 01:38:12.109602 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.109574    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t58fj_8d0f282b-d478-4315-b903-74f767be07ae/init-textfile/0.log"
Aug 01 01:38:12.219720 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.219629    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-7htgc_bf82be8f-4cfa-4545-aeaf-76d8ecb0d900/kube-rbac-proxy-main/0.log"
Aug 01 01:38:12.241950 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.241910    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-7htgc_bf82be8f-4cfa-4545-aeaf-76d8ecb0d900/kube-rbac-proxy-self/0.log"
Aug 01 01:38:12.268027 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.267994    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-7htgc_bf82be8f-4cfa-4545-aeaf-76d8ecb0d900/openshift-state-metrics/0.log"
Aug 01 01:38:12.488170 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.488051    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-54k48_a1a483c3-c11e-44e0-9ccf-dbd441ea42fb/prometheus-operator/0.log"
Aug 01 01:38:12.509473 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.509448    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-54k48_a1a483c3-c11e-44e0-9ccf-dbd441ea42fb/kube-rbac-proxy/0.log"
Aug 01 01:38:12.534332 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:12.534298    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-66f5h_eeb95623-3f3a-40e6-8145-ddecda8ff61a/prometheus-operator-admission-webhook/0.log"
Aug 01 01:38:15.820446 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:15.820407    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"]
Aug 01 01:38:15.828692 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:15.828652    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:15.836004 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:15.835972    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"]
Aug 01 01:38:15.985237 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:15.985200    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-lib-modules\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:15.985427 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:15.985262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-podres\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:15.985427 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:15.985303    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-proc\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:15.985427 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:15.985356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-sys\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:15.985427 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:15.985388    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpn8\" (UniqueName: \"kubernetes.io/projected/edda33fd-7760-4485-8e13-51909c4a079a-kube-api-access-vrpn8\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.086080 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.085998    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-sys\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.086335 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.086312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpn8\" (UniqueName: \"kubernetes.io/projected/edda33fd-7760-4485-8e13-51909c4a079a-kube-api-access-vrpn8\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.086497 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.086477    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-lib-modules\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.086627 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.086612    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-podres\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.086762 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.086749    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-proc\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.086980 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.086962    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-proc\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.087163 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.087145    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-sys\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.087701 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.087674    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-lib-modules\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.087986 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.087915    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edda33fd-7760-4485-8e13-51909c4a079a-podres\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.096148 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.096124    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpn8\" (UniqueName: \"kubernetes.io/projected/edda33fd-7760-4485-8e13-51909c4a079a-kube-api-access-vrpn8\") pod \"perf-node-gather-daemonset-dcksj\" (UID: \"edda33fd-7760-4485-8e13-51909c4a079a\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.142780 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.142680    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:16.323298 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:16.322520    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"]
Aug 01 01:38:17.102477 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:17.101995    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj" event={"ID":"edda33fd-7760-4485-8e13-51909c4a079a","Type":"ContainerStarted","Data":"227f28a3f64e2db4ba63fcedaff1d3f2e9cef9d931f4b4af2430f17ef316e3f5"}
Aug 01 01:38:17.102477 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:17.102064    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj" event={"ID":"edda33fd-7760-4485-8e13-51909c4a079a","Type":"ContainerStarted","Data":"fcd869b0fa50025e24afa95fc63a5dfa3d17327e27ba98f287ae1709d34dfc35"}
Aug 01 01:38:17.102477 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:17.102269    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:17.122362 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:17.122301    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj" podStartSLOduration=2.122277505 podStartE2EDuration="2.122277505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 01:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:38:17.120520057 +0000 UTC m=+3863.354452326" watchObservedRunningTime="2026-08-01 01:38:17.122277505 +0000 UTC m=+3863.356209772"
Aug 01 01:38:17.408885 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:17.408806    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r7rdc_d36e3319-9c8a-480e-aa41-4e7a98c0532a/dns/0.log"
Aug 01 01:38:17.427507 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:17.427481    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r7rdc_d36e3319-9c8a-480e-aa41-4e7a98c0532a/kube-rbac-proxy/0.log"
Aug 01 01:38:17.533120 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:17.533070    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k8jhp_ad989299-a180-4b5d-ae39-9a5ffc294368/dns-node-resolver/0.log"
Aug 01 01:38:18.120595 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:18.120561    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5fvlj_85d6afc8-b208-4fcf-b770-47f6560962a3/node-ca/0.log"
Aug 01 01:38:19.236036 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:19.236008    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7798dfdbd7-l7cgg_4212e104-5c9d-49b3-8fca-3e69f1413f3a/kube-auth-proxy/0.log"
Aug 01 01:38:19.328128 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:19.328101    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-ld4rx_3020d525-e75f-4340-b6c2-e28a56c409b4/payload-pre-processing/0.log"
Aug 01 01:38:20.164344 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:20.164315    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5w4jn_a8d42714-2762-4696-8f20-c1005f745797/serve-healthcheck-canary/0.log"
Aug 01 01:38:20.826617 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:20.826573    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4qrs_c91d0d2b-a7e4-4d7c-b252-86e77bd86964/kube-rbac-proxy/0.log"
Aug 01 01:38:20.847240 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:20.847209    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4qrs_c91d0d2b-a7e4-4d7c-b252-86e77bd86964/exporter/0.log"
Aug 01 01:38:20.870737 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:20.870711    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4qrs_c91d0d2b-a7e4-4d7c-b252-86e77bd86964/extractor/0.log"
Aug 01 01:38:23.123486 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:23.123454    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-dcksj"
Aug 01 01:38:23.358543 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:23.358493    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-787d5bc65-p4m2m_5778d33b-8872-4191-96ab-c210256d3c25/manager/0.log"
Aug 01 01:38:23.377192 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:23.377117    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2npxt_7f8797bf-066c-4c26-9ffd-5d1382eee2b5/server/0.log"
Aug 01 01:38:23.625305 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:23.625259    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f88dcc48-cm5hv_5c2ef3b9-05f2-47bb-ac1b-6f64858c0874/manager/0.log"
Aug 01 01:38:23.641858 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:23.641794    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f88dcc48-cm5hv_5c2ef3b9-05f2-47bb-ac1b-6f64858c0874/copy-manifests/0.log"
Aug 01 01:38:24.901585 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:24.901554    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8588bf7fd6-hr66k_2d50c4ac-3b53-4c30-9638-579fef54b213/manager/0.log"
Aug 01 01:38:26.296572 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:26.296501    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7798dfdbd7-l7cgg_4212e104-5c9d-49b3-8fca-3e69f1413f3a/kube-auth-proxy/0.log"
Aug 01 01:38:26.352991 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:26.352917    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-ld4rx_3020d525-e75f-4340-b6c2-e28a56c409b4/payload-pre-processing/0.log"
Aug 01 01:38:27.952215 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:27.952143    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7798dfdbd7-l7cgg_4212e104-5c9d-49b3-8fca-3e69f1413f3a/kube-auth-proxy/0.log"
Aug 01 01:38:28.014053 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:28.013981    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-ld4rx_3020d525-e75f-4340-b6c2-e28a56c409b4/payload-pre-processing/0.log"
Aug 01 01:38:30.026466 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:30.026433    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-ck962_bb9510cb-97a2-4d67-9ffe-0d0902e6bf04/migrator/0.log"
Aug 01 01:38:30.047518 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:30.047493    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-ck962_bb9510cb-97a2-4d67-9ffe-0d0902e6bf04/graceful-termination/0.log"
Aug 01 01:38:31.327607 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.327571    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-59ckb_4efd71b3-8bef-4b32-ae27-5d1e83c2d393/kube-multus/0.log"
Aug 01 01:38:31.684690 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.684616    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tq6z2_c2ff6e14-3a7a-4d02-a3c1-5d4e34712996/kube-multus-additional-cni-plugins/0.log"
Aug 01 01:38:31.705511 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.705484    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tq6z2_c2ff6e14-3a7a-4d02-a3c1-5d4e34712996/egress-router-binary-copy/0.log"
Aug 01 01:38:31.725498 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.725472    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tq6z2_c2ff6e14-3a7a-4d02-a3c1-5d4e34712996/cni-plugins/0.log"
Aug 01 01:38:31.747796 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.747772    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tq6z2_c2ff6e14-3a7a-4d02-a3c1-5d4e34712996/bond-cni-plugin/0.log"
Aug 01 01:38:31.768247 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.768228    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tq6z2_c2ff6e14-3a7a-4d02-a3c1-5d4e34712996/routeoverride-cni/0.log"
Aug 01 01:38:31.791491 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.791469    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tq6z2_c2ff6e14-3a7a-4d02-a3c1-5d4e34712996/whereabouts-cni-bincopy/0.log"
Aug 01 01:38:31.810302 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.810279    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tq6z2_c2ff6e14-3a7a-4d02-a3c1-5d4e34712996/whereabouts-cni/0.log"
Aug 01 01:38:31.947283 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.947213    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-99f8t_31926079-d63f-43de-b52c-ced2e283ba10/network-metrics-daemon/0.log"
Aug 01 01:38:31.964669 ip-10-0-141-142 kubenswrapper[2570]: I0801 01:38:31.964645    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-99f8t_31926079-d63f-43de-b52c-ced2e283ba10/kube-rbac-proxy/0.log"